[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:12:43 dist-test-slave-pswb krb5kdc[16707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:12:43 dist-test-slave-pswb krb5kdc[16707](info): set up 2 sockets
Jun 20 23:12:43 dist-test-slave-pswb krb5kdc[16707](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 23:12:44.860641 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:12:44 dist-test-slave-pswb krb5kdc[16707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:12:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:12:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:12:44.976397 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:46367
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:46367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:45.053679 16720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:45.053892 16720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:45.053925 16720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:45.055004 16720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:12:45.055069 16720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:45.055094 16720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:45.055117 16720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:12:45.055160 16720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:12:45.056576 16720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:46367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:46367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:45.056871 16720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:45.057149 16720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:45.062487 16728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:45.062520 16725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:45.062585 16726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:45.062588 16720 server_base.cc:1034] running on GCE node
I20260620 23:12:45.062925 16720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:45.063203 16720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:45.064375 16720 hybrid_clock.cc:648] HybridClock initialized: now 1781997165064353 us; error 54 us; skew 500 ppm
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[16707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997165, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:45.066699 16720 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:12:45.067137 16720 webserver.cc:460] Webserver started at http://127.16.79.62:45439/ using document root <none> and password file <none>
I20260620 23:12:45.067375 16720 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:45.067443 16720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:45.067591 16720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:45.068553 16720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "8f5aa91af73a4744afe477f8eb0f357e"
format_stamp: "Formatted at 2026-06-20 23:12:45 on dist-test-slave-pswb"
server_key: "407a424890f9038ee46341adc6ba3eb4"
server_key_iv: "b949c31d6a0d91aca8587c6e9e489fbc"
server_key_version: "encryptionkey@0"
I20260620 23:12:45.068933 16720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "8f5aa91af73a4744afe477f8eb0f357e"
format_stamp: "Formatted at 2026-06-20 23:12:45 on dist-test-slave-pswb"
server_key: "407a424890f9038ee46341adc6ba3eb4"
server_key_iv: "b949c31d6a0d91aca8587c6e9e489fbc"
server_key_version: "encryptionkey@0"
I20260620 23:12:45.070286 16720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:45.071411 16735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:45.071623 16720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:45.071705 16720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "8f5aa91af73a4744afe477f8eb0f357e"
format_stamp: "Formatted at 2026-06-20 23:12:45 on dist-test-slave-pswb"
server_key: "407a424890f9038ee46341adc6ba3eb4"
server_key_iv: "b949c31d6a0d91aca8587c6e9e489fbc"
server_key_version: "encryptionkey@0"
I20260620 23:12:45.071796 16720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:45.096650 16720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:45.098889 16720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:45.099056 16720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:45.103470 16720 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:46367
I20260620 23:12:45.103521 16787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:46367 every 8 connection(s)
I20260620 23:12:45.103848 16720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:12:45.104660 16788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:45.106823 16788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e: Bootstrap starting.
I20260620 23:12:45.107334 16788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:45.107563 16788 log.cc:824] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:45.108204 16788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e: No bootstrap required, opened a new log
I20260620 23:12:45.109803 16788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5aa91af73a4744afe477f8eb0f357e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46367 } }
I20260620 23:12:45.109933 16788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:45.109956 16788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5aa91af73a4744afe477f8eb0f357e, State: Initialized, Role: FOLLOWER
I20260620 23:12:45.110085 16788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5aa91af73a4744afe477f8eb0f357e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46367 } }
I20260620 23:12:45.110167 16788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:12:45.110188 16788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:12:45.110215 16788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:45.110666 16788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5aa91af73a4744afe477f8eb0f357e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46367 } }
I20260620 23:12:45.110769 16788 leader_election.cc:302] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f5aa91af73a4744afe477f8eb0f357e; no voters:
I20260620 23:12:45.110965 16788 leader_election.cc:288] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:12:45.111079 16791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:45.111219 16788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:12:45.111166 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 16720
I20260620 23:12:45.111338 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:12:45.111340 16791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [term 1 LEADER]: Becoming Leader. State: Replica: 8f5aa91af73a4744afe477f8eb0f357e, State: Running, Role: LEADER
I20260620 23:12:45.111438 16791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5aa91af73a4744afe477f8eb0f357e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46367 } }
I20260620 23:12:45.111909 16792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f5aa91af73a4744afe477f8eb0f357e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5aa91af73a4744afe477f8eb0f357e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46367 } } }
I20260620 23:12:45.111928 16793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f5aa91af73a4744afe477f8eb0f357e. Latest consensus state: current_term: 1 leader_uuid: "8f5aa91af73a4744afe477f8eb0f357e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5aa91af73a4744afe477f8eb0f357e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46367 } } }
I20260620 23:12:45.112007 16792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:45.112035 16793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:45.112269 16700 external_mini_cluster.cc:1428] Setting key 6a506862bad329a4ce496b87ec90149e
I20260620 23:12:45.112350 16800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:12:45.112962 16800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:12:45.117033 16800 catalog_manager.cc:1347] Generated new cluster ID: 3cea9efa5dcc4cc4b8c3037cc68a9a47
I20260620 23:12:45.117097 16800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[16707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:45.131074 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:45.113643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.16.79.62:46367)
0620 23:12:45.116074 (+ 2431us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:45.116082 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:45.116110 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:12:45.116212 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:45.116217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:45.116281 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:45.116628 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:45.118747 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.120200 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.121764 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.122360 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.122474 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:45.126673 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:45.126677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:45.126678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:45.126688 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:45.128922 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:45.129426 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:45.129429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:45.129429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:45.129533 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:45.129817 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:45.129817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:45.129818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:45.130106 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:45.130250 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:45.130549 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 23:12:45.130729 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2384,"thread_start_us":81,"threads_started":1}
I20260620 23:12:45.150981 16800 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:12:45.151513 16800 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:12:45.158916 16800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f5aa91af73a4744afe477f8eb0f357e: Generated new TSK 0
I20260620 23:12:45.159117 16800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:12:45.188416 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46367
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:45.268996 16814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:45.269181 16814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:45.269204 16814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:45.270309 16814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:45.270349 16814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:45.270375 16814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:12:45.271800 16814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46367
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:45.272074 16814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:45.272300 16814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:45.277335 16822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:45.277318 16819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:45.277377 16814 server_base.cc:1034] running on GCE node
W20260620 23:12:45.277313 16820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:45.277729 16814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:45.277966 16814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:45.279132 16814 hybrid_clock.cc:648] HybridClock initialized: now 1781997165279129 us; error 33 us; skew 500 ppm
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[16707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997165, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:45.281661 16814 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:12:45.282203 16814 webserver.cc:460] Webserver started at http://127.16.79.1:43569/ using document root <none> and password file <none>
I20260620 23:12:45.282459 16814 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:45.282526 16814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:45.282658 16814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:45.283628 16814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "d062a979855d40959ba8761fe3e87411"
format_stamp: "Formatted at 2026-06-20 23:12:45 on dist-test-slave-pswb"
server_key: "402d9befb9e4369dbef2f19956a06fdd"
server_key_iv: "28821ae8d5b1c6997f804bb9920dd99b"
server_key_version: "encryptionkey@0"
I20260620 23:12:45.283978 16814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "d062a979855d40959ba8761fe3e87411"
format_stamp: "Formatted at 2026-06-20 23:12:45 on dist-test-slave-pswb"
server_key: "402d9befb9e4369dbef2f19956a06fdd"
server_key_iv: "28821ae8d5b1c6997f804bb9920dd99b"
server_key_version: "encryptionkey@0"
I20260620 23:12:45.285331 16814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:12:45.286082 16829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:45.286254 16814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:12:45.286346 16814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "d062a979855d40959ba8761fe3e87411"
format_stamp: "Formatted at 2026-06-20 23:12:45 on dist-test-slave-pswb"
server_key: "402d9befb9e4369dbef2f19956a06fdd"
server_key_iv: "28821ae8d5b1c6997f804bb9920dd99b"
server_key_version: "encryptionkey@0"
I20260620 23:12:45.286422 16814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:45.303170 16814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:45.305588 16814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:45.305802 16814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:45.306100 16814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:45.306484 16814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:45.306546 16814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:45.306594 16814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:45.306628 16814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:45.313963 16814 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:33029
I20260620 23:12:45.313997 16942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:33029 every 8 connection(s)
I20260620 23:12:45.314313 16814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[16707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997165, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:45.323354 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 16814
I20260620 23:12:45.323467 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:12:45.323632 16700 external_mini_cluster.cc:1428] Setting key 6a07b1c593ce1cb794d8dbb37c8a45f7
I20260620 23:12:45.323808 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:45.315047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:47879 (local address 127.16.79.62:46367)
0620 23:12:45.315090 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:45.315092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:45.315236 (+ 144us) server_negotiation.cc:407] Connection header received
0620 23:12:45.315735 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:45.315736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:45.315750 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:45.315807 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:45.316860 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.317378 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.318352 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.318558 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.318623 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:45.320627 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:45.320628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:45.320628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:45.320634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:45.322088 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:45.322734 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:45.322735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:45.322736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:45.322791 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:45.323115 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:45.323116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:45.323116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:45.323229 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:45.323323 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:45.323636 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 23:12:45.323713 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:12:45.324010 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:45.314964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46367 (local address 127.16.79.1:47879)
0620 23:12:45.315125 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:45.315164 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:45.315608 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:45.315878 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:45.315881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:45.316445 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:45.316628 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:45.316632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.317473 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.317474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:45.318254 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:45.318255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.318661 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.318662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:45.318769 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:45.319268 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:45.319276 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:45.320532 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:45.322225 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:45.322236 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:45.322239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:45.322606 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:45.322893 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:45.322894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:45.322894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:45.323026 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:45.323326 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:45.323333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:45.323484 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 23:12:45.323613 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 23:12:45.323783 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260620 23:12:45.324595 16943 heartbeater.cc:347] Connected to a master server at 127.16.79.62:46367
I20260620 23:12:45.324671 16943 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:45.324874 16943 heartbeater.cc:511] Master 127.16.79.62:46367 requested a full tablet report, sending...
I20260620 23:12:45.325394 16752 ts_manager.cc:194] Registered new tserver with Master: d062a979855d40959ba8761fe3e87411 (127.16.79.1:33029)
I20260620 23:12:45.326305 16752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:47879
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:12:45.349009 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46367
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:45.428920 16948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:45.429127 16948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:45.429163 16948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:45.430295 16948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:45.430357 16948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:45.430402 16948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:12:45.431805 16948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46367
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:45.432049 16948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:45.432277 16948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:45.437101 16956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:45.437096 16953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:45.437105 16954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:45.437325 16948 server_base.cc:1034] running on GCE node
I20260620 23:12:45.437534 16948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:45.437774 16948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:45.438931 16948 hybrid_clock.cc:648] HybridClock initialized: now 1781997165438929 us; error 38 us; skew 500 ppm
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[16707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997165, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:45.441315 16948 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:12:45.441725 16948 webserver.cc:460] Webserver started at http://127.16.79.2:40341/ using document root <none> and password file <none>
I20260620 23:12:45.441891 16948 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:45.441929 16948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:45.442011 16948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:45.442909 16948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "e8bd236a7be34856a0a56380225b4d4f"
format_stamp: "Formatted at 2026-06-20 23:12:45 on dist-test-slave-pswb"
server_key: "295c627e0015552d4c7a7569ab4c04bd"
server_key_iv: "224c3dee939afbf19c8613f6f295b504"
server_key_version: "encryptionkey@0"
I20260620 23:12:45.443208 16948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "e8bd236a7be34856a0a56380225b4d4f"
format_stamp: "Formatted at 2026-06-20 23:12:45 on dist-test-slave-pswb"
server_key: "295c627e0015552d4c7a7569ab4c04bd"
server_key_iv: "224c3dee939afbf19c8613f6f295b504"
server_key_version: "encryptionkey@0"
I20260620 23:12:45.444446 16948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:12:45.445133 16963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:45.445288 16948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:45.445377 16948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "e8bd236a7be34856a0a56380225b4d4f"
format_stamp: "Formatted at 2026-06-20 23:12:45 on dist-test-slave-pswb"
server_key: "295c627e0015552d4c7a7569ab4c04bd"
server_key_iv: "224c3dee939afbf19c8613f6f295b504"
server_key_version: "encryptionkey@0"
I20260620 23:12:45.445446 16948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:45.461923 16948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:45.463658 16948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:45.463771 16948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:45.464010 16948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:45.464284 16948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:45.464337 16948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:45.464376 16948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:45.464403 16948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:45.470158 16948 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:41379
I20260620 23:12:45.470228 17076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:41379 every 8 connection(s)
I20260620 23:12:45.470516 16948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:12:45.473206 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 16948
I20260620 23:12:45.473301 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:12:45.473445 16700 external_mini_cluster.cc:1428] Setting key 037648542a3f7f0766505f4381662e97
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[16707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997165, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:45.480163 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:45.471207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:39425 (local address 127.16.79.62:46367)
0620 23:12:45.471271 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:45.471274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:45.471452 (+ 178us) server_negotiation.cc:407] Connection header received
0620 23:12:45.471854 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:45.471855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:45.471864 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:45.471909 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:45.472831 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.473576 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.474691 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.475020 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.475109 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:45.477410 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:45.477412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:45.477413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:45.477421 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:45.478817 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:45.479216 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:45.479218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:45.479218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:45.479286 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:45.479540 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:45.479541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:45.479541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:45.479647 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:45.479719 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:45.479998 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 23:12:45.480076 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:12:45.480372 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:45.471181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46367 (local address 127.16.79.2:39425)
0620 23:12:45.471329 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:45.471359 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:45.471777 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:45.471965 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:45.471968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:45.472495 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:45.472660 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:45.472665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.473676 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.473677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:45.474609 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:45.474612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.475114 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.475115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:45.475221 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:45.475784 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:45.475795 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:45.477318 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:45.478916 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:45.478922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:45.478925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:45.479139 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:45.479355 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:45.479356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:45.479357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:45.479466 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:45.479723 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:45.479736 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:45.479867 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 23:12:45.479974 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 23:12:45.480117 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260620 23:12:45.480852 17077 heartbeater.cc:347] Connected to a master server at 127.16.79.62:46367
I20260620 23:12:45.480944 17077 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:45.481117 17077 heartbeater.cc:511] Master 127.16.79.62:46367 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
I20260620 23:12:45.481775 16752 ts_manager.cc:194] Registered new tserver with Master: e8bd236a7be34856a0a56380225b4d4f (127.16.79.2:41379)
I20260620 23:12:45.482095 16752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:39425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:12:45.499858 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46367
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:45.577466 17082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:45.577648 17082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:45.577672 17082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:45.578703 17082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:45.578737 17082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:45.578763 17082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:12:45.580071 17082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46367
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:45.580230 17082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:45.580384 17082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:45.585062 17087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:45.585175 17082 server_base.cc:1034] running on GCE node
W20260620 23:12:45.585088 17088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:45.585096 17090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:45.585492 17082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:45.585731 17082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:45.586900 17082 hybrid_clock.cc:648] HybridClock initialized: now 1781997165586879 us; error 30 us; skew 500 ppm
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[16707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997165, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:45.589365 17082 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:12:45.589792 17082 webserver.cc:460] Webserver started at http://127.16.79.3:42669/ using document root <none> and password file <none>
I20260620 23:12:45.590013 17082 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:45.590080 17082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:45.590229 17082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:45.591220 17082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "303cb734b5294bcc8fa6fbf9715dfafa"
format_stamp: "Formatted at 2026-06-20 23:12:45 on dist-test-slave-pswb"
server_key: "29633e3688209dd6943130e8f5ea0d1a"
server_key_iv: "8a4c57535026dbf547091cf9abe3929b"
server_key_version: "encryptionkey@0"
I20260620 23:12:45.591502 17082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "303cb734b5294bcc8fa6fbf9715dfafa"
format_stamp: "Formatted at 2026-06-20 23:12:45 on dist-test-slave-pswb"
server_key: "29633e3688209dd6943130e8f5ea0d1a"
server_key_iv: "8a4c57535026dbf547091cf9abe3929b"
server_key_version: "encryptionkey@0"
I20260620 23:12:45.592669 17082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:12:45.593401 17097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:45.593547 17082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:12:45.593617 17082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "303cb734b5294bcc8fa6fbf9715dfafa"
format_stamp: "Formatted at 2026-06-20 23:12:45 on dist-test-slave-pswb"
server_key: "29633e3688209dd6943130e8f5ea0d1a"
server_key_iv: "8a4c57535026dbf547091cf9abe3929b"
server_key_version: "encryptionkey@0"
I20260620 23:12:45.593667 17082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:45.609002 17082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:45.610716 17082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:45.610844 17082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:45.611078 17082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:45.611306 17082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:45.611335 17082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:45.611354 17082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:45.611367 17082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:45.617252 17082 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:43031
I20260620 23:12:45.617304 17210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:43031 every 8 connection(s)
I20260620 23:12:45.617596 17082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[16707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997165, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:45.624006 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 17082
I20260620 23:12:45.624117 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:12:45.624337 16700 external_mini_cluster.cc:1428] Setting key 0349141ca20ab7fcbe1b1ac2dfc02730
I20260620 23:12:45.626756 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:45.618294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:49617 (local address 127.16.79.62:46367)
0620 23:12:45.618363 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:45.618365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:45.618538 (+ 173us) server_negotiation.cc:407] Connection header received
0620 23:12:45.618957 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:45.618957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:45.618967 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:45.619017 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:45.619849 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.620327 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.621297 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.621475 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.621530 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:45.623489 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:45.623490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:45.623490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:45.623496 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:45.624979 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:45.625542 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:45.625543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:45.625543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:45.625610 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:45.625899 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:45.625900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:45.625900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:45.626001 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:45.626057 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:45.626635 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 23:12:45.626697 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:12:45.627013 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:45.618274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46367 (local address 127.16.79.3:49617)
0620 23:12:45.618424 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:45.618456 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:45.618880 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:45.619076 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:45.619079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:45.619605 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:45.619778 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:45.619783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.620400 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.620401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:45.621221 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:45.621223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.621548 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.621549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:45.621635 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:45.622096 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:45.622103 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:45.623404 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:45.625071 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:45.625077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:45.625080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:45.625349 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:45.625696 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:45.625697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:45.625698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:45.625826 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:45.626146 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:45.626160 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:45.626474 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 23:12:45.626612 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 23:12:45.626735 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260620 23:12:45.627508 17211 heartbeater.cc:347] Connected to a master server at 127.16.79.62:46367
I20260620 23:12:45.627583 17211 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:45.627759 17211 heartbeater.cc:511] Master 127.16.79.62:46367 requested a full tablet report, sending...
I20260620 23:12:45.628141 16752 ts_manager.cc:194] Registered new tserver with Master: 303cb734b5294bcc8fa6fbf9715dfafa (127.16.79.3:43031)
I20260620 23:12:45.628502 16752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:49617
I20260620 23:12:45.636569 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[16707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[16707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.79.1@KRBTEST.COM
I20260620 23:12:45.657940 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:45.650751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.16.79.1:33029)
0620 23:12:45.650912 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:45.650916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:45.650925 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:12:45.650947 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:45.650949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:45.650961 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:45.651024 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:45.651535 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.652238 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.652989 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.653275 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.653330 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:45.655356 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:45.655362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:45.655363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:45.655369 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:45.656988 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:45.657288 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:45.657290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:45.657290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:45.657341 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:45.657495 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:45.657496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:45.657496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:45.657604 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:45.657706 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:45.657763 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 23:12:45.657818 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
W20260620 23:12:45.658286 16857 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47892
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[16707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[16707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.1@KRBTEST.COM
I20260620 23:12:45.679100 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:45.671959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.16.79.1:33029)
0620 23:12:45.672025 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:45.672027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:45.672126 (+ 99us) server_negotiation.cc:407] Connection header received
0620 23:12:45.672212 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:45.672213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:45.672225 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:45.672303 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:45.672882 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.673391 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.674135 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:45.674352 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:45.674409 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:45.676548 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:45.676549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:45.676549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:45.676555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:45.677967 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:45.678303 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:45.678303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:45.678304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:45.678352 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:45.678570 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:45.678571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:45.678571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:45.678651 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:45.678743 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:45.678894 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 23:12:45.679021 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:12:45.679890 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 16814
I20260620 23:12:45.685596 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 16948
I20260620 23:12:45.690284 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 17082
I20260620 23:12:45.694336 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 16720
2026-06-20T23:12:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1893 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[17225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[17225](info): set up 2 sockets
Jun 20 23:12:45 dist-test-slave-pswb krb5kdc[17225](info): commencing operation
krb5kdc: starting...
W20260620 23:12:46.732203 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:12:46 dist-test-slave-pswb krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:12:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:12:46.845522 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:43941
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:43941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:46.923787 17238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:46.924010 17238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:46.924045 17238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:46.925194 17238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:12:46.925261 17238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:46.925288 17238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:46.925321 17238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:12:46.925357 17238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:12:46.926796 17238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:43941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:43941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:46.927062 17238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:46.927285 17238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:46.932350 17243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:46.932439 17246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:46.932365 17244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:46.932781 17238 server_base.cc:1034] running on GCE node
I20260620 23:12:46.933020 17238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:46.933277 17238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:46.934448 17238 hybrid_clock.cc:648] HybridClock initialized: now 1781997166934426 us; error 31 us; skew 500 ppm
Jun 20 23:12:46 dist-test-slave-pswb krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997166, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:46.936895 17238 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:12:46.937343 17238 webserver.cc:460] Webserver started at http://127.16.79.62:40655/ using document root <none> and password file <none>
I20260620 23:12:46.937559 17238 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:46.937626 17238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:46.937770 17238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:46.938753 17238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "1e635235e4944c02a9b366bcb29331e3"
format_stamp: "Formatted at 2026-06-20 23:12:46 on dist-test-slave-pswb"
server_key: "09ceec416a79a2c8dc945ba5479e9485"
server_key_iv: "12dcfda33bef603140bf8bdb41e780a0"
server_key_version: "encryptionkey@0"
I20260620 23:12:46.939113 17238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "1e635235e4944c02a9b366bcb29331e3"
format_stamp: "Formatted at 2026-06-20 23:12:46 on dist-test-slave-pswb"
server_key: "09ceec416a79a2c8dc945ba5479e9485"
server_key_iv: "12dcfda33bef603140bf8bdb41e780a0"
server_key_version: "encryptionkey@0"
I20260620 23:12:46.940377 17238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:12:46.941170 17253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:46.941349 17238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:12:46.941448 17238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "1e635235e4944c02a9b366bcb29331e3"
format_stamp: "Formatted at 2026-06-20 23:12:46 on dist-test-slave-pswb"
server_key: "09ceec416a79a2c8dc945ba5479e9485"
server_key_iv: "12dcfda33bef603140bf8bdb41e780a0"
server_key_version: "encryptionkey@0"
I20260620 23:12:46.941519 17238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:46.950502 17238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:46.952644 17238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:46.952818 17238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:46.956470 17238 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:43941
I20260620 23:12:46.956499 17305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:43941 every 8 connection(s)
I20260620 23:12:46.956884 17238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:12:46.957713 17306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:46.959759 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 17238
I20260620 23:12:46.959853 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:12:46.960002 16700 external_mini_cluster.cc:1428] Setting key 23e4c66b405388e2f6be718f6db4beaf
I20260620 23:12:46.960002 17306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3: Bootstrap starting.
I20260620 23:12:46.960633 17306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:46.960944 17306 log.cc:824] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:46.961572 17306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3: No bootstrap required, opened a new log
I20260620 23:12:46.963315 17306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e635235e4944c02a9b366bcb29331e3" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 43941 } }
I20260620 23:12:46.963461 17306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:46.963507 17306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e635235e4944c02a9b366bcb29331e3, State: Initialized, Role: FOLLOWER
I20260620 23:12:46.963639 17306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e635235e4944c02a9b366bcb29331e3" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 43941 } }
I20260620 23:12:46.963733 17306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:12:46.963774 17306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:12:46.963822 17306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:46.964452 17306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e635235e4944c02a9b366bcb29331e3" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 43941 } }
I20260620 23:12:46.964560 17306 leader_election.cc:302] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e635235e4944c02a9b366bcb29331e3; no voters:
I20260620 23:12:46.964721 17306 leader_election.cc:288] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:12:46.964749 17311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:46.964865 17311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [term 1 LEADER]: Becoming Leader. State: Replica: 1e635235e4944c02a9b366bcb29331e3, State: Running, Role: LEADER
I20260620 23:12:46.964932 17311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e635235e4944c02a9b366bcb29331e3" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 43941 } }
I20260620 23:12:46.964968 17306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:12:46.965210 17311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e635235e4944c02a9b366bcb29331e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e635235e4944c02a9b366bcb29331e3" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 43941 } } }
I20260620 23:12:46.965273 17311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:46.965250 17313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e635235e4944c02a9b366bcb29331e3. Latest consensus state: current_term: 1 leader_uuid: "1e635235e4944c02a9b366bcb29331e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e635235e4944c02a9b366bcb29331e3" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 43941 } } }
I20260620 23:12:46.965358 17313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:46.966109 17315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:12:46.966632 17315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:12:46 dist-test-slave-pswb krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:46.968222 17315 catalog_manager.cc:1347] Generated new cluster ID: c1cc95372e8548a8896ee589deb256a2
I20260620 23:12:46.968277 17315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:12:46.971338 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:46.960482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.16.79.62:43941)
0620 23:12:46.960670 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:46.960679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:46.960726 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:12:46.961224 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:46.961231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:46.961273 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:46.961414 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:46.962101 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:46.962984 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:46.963859 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:46.964458 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:46.964578 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:46.967756 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:46.967760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:46.967760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:46.967771 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:46.969755 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:46.970140 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:46.970143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:46.970143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:46.970247 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:46.970441 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:46.970442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:46.970443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:46.970743 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:46.970891 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:46.971004 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 23:12:46.971134 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260620 23:12:46.976178 17315 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:12:46.976857 17315 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:12:46.984151 17315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e635235e4944c02a9b366bcb29331e3: Generated new TSK 0
I20260620 23:12:46.984300 17315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:12:47.015964 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:43941
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:47.094735 17332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:47.094954 17332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:47.094993 17332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:47.096124 17332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:47.096189 17332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:47.096235 17332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:12:47.097754 17332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:43941
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:47.098013 17332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:47.098235 17332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:47.103017 17338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:47.103041 17340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:47.103017 17337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:47.103320 17332 server_base.cc:1034] running on GCE node
I20260620 23:12:47.103536 17332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:47.103724 17332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:47.104861 17332 hybrid_clock.cc:648] HybridClock initialized: now 1781997167104849 us; error 30 us; skew 500 ppm
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997167, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:47.107241 17332 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:12:47.107669 17332 webserver.cc:460] Webserver started at http://127.16.79.1:39727/ using document root <none> and password file <none>
I20260620 23:12:47.107856 17332 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:47.107895 17332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:47.107977 17332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:47.108939 17332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "a2ffacce26c14a7990933a124a985edc"
format_stamp: "Formatted at 2026-06-20 23:12:47 on dist-test-slave-pswb"
server_key: "77fefd4d12c4fd6cd5529cb4133aa715"
server_key_iv: "333b32ce71a309ea4b709f33fc4bdf82"
server_key_version: "encryptionkey@0"
I20260620 23:12:47.109220 17332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "a2ffacce26c14a7990933a124a985edc"
format_stamp: "Formatted at 2026-06-20 23:12:47 on dist-test-slave-pswb"
server_key: "77fefd4d12c4fd6cd5529cb4133aa715"
server_key_iv: "333b32ce71a309ea4b709f33fc4bdf82"
server_key_version: "encryptionkey@0"
I20260620 23:12:47.110563 17332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:47.111738 17347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:47.112015 17332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:47.112104 17332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "a2ffacce26c14a7990933a124a985edc"
format_stamp: "Formatted at 2026-06-20 23:12:47 on dist-test-slave-pswb"
server_key: "77fefd4d12c4fd6cd5529cb4133aa715"
server_key_iv: "333b32ce71a309ea4b709f33fc4bdf82"
server_key_version: "encryptionkey@0"
I20260620 23:12:47.112212 17332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:47.124411 17332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:47.126813 17332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:47.126968 17332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:47.127215 17332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:47.127936 17332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:47.127987 17332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:47.128180 17332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:47.128228 17332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:47.135275 17332 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:33877
I20260620 23:12:47.135326 17460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:33877 every 8 connection(s)
I20260620 23:12:47.135643 17332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:12:47.140758 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 17332
I20260620 23:12:47.140856 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:12:47.141023 16700 external_mini_cluster.cc:1428] Setting key 5dd4d76738eed746ff78b69e39108d3f
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997167, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:47.146107 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.136400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:56741 (local address 127.16.79.62:43941)
0620 23:12:47.136441 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:47.136443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:47.136645 (+ 202us) server_negotiation.cc:407] Connection header received
0620 23:12:47.137755 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:47.137757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:47.137769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:47.137814 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:47.138405 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.138913 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.139892 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.140112 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.140195 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.143250 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:47.143252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:47.143252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:47.143260 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:47.144672 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.145160 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.145161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.145161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.145223 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.145475 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.145476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.145477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.145589 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:47.145663 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:47.145969 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 23:12:47.146044 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:12:47.146373 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.136371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:43941 (local address 127.16.79.1:56741)
0620 23:12:47.136518 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:47.136549 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:47.137536 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:47.137885 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:47.137889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:47.138129 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:47.138307 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.138311 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.139006 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.139007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.139793 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.139795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.140211 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.140212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.140333 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.140990 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:47.140997 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:47.143090 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:47.144799 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:47.144804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:47.144806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:47.145076 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:47.145289 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:47.145290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:47.145290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:47.145407 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:47.145656 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:47.145661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:47.145823 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 23:12:47.145938 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 23:12:47.146112 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260620 23:12:47.146878 17461 heartbeater.cc:347] Connected to a master server at 127.16.79.62:43941
I20260620 23:12:47.146967 17461 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:47.147135 17461 heartbeater.cc:511] Master 127.16.79.62:43941 requested a full tablet report, sending...
I20260620 23:12:47.147831 17270 ts_manager.cc:194] Registered new tserver with Master: a2ffacce26c14a7990933a124a985edc (127.16.79.1:33877)
I20260620 23:12:47.148524 17270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:56741
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:12:47.168619 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:43941
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:47.250039 17466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:47.250265 17466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:47.250299 17466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:47.251441 17466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:47.251513 17466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:47.251559 17466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:12:47.253547 17466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:43941
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:47.253799 17466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:47.254040 17466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:47.258741 17474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:47.258741 17472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:47.258884 17466 server_base.cc:1034] running on GCE node
W20260620 23:12:47.258796 17471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:47.259167 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:47.259406 17466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:47.260540 17466 hybrid_clock.cc:648] HybridClock initialized: now 1781997167260520 us; error 39 us; skew 500 ppm
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997167, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:47.262924 17466 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:12:47.263324 17466 webserver.cc:460] Webserver started at http://127.16.79.2:36781/ using document root <none> and password file <none>
I20260620 23:12:47.263535 17466 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:47.263577 17466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:47.263710 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:47.264933 17466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "4526ace6898b43db97060eca562e3071"
format_stamp: "Formatted at 2026-06-20 23:12:47 on dist-test-slave-pswb"
server_key: "833ee37eca7fe04af30c561718ef5f31"
server_key_iv: "768c1478eb7533c853ef73b6a8bcbcf0"
server_key_version: "encryptionkey@0"
I20260620 23:12:47.265270 17466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "4526ace6898b43db97060eca562e3071"
format_stamp: "Formatted at 2026-06-20 23:12:47 on dist-test-slave-pswb"
server_key: "833ee37eca7fe04af30c561718ef5f31"
server_key_iv: "768c1478eb7533c853ef73b6a8bcbcf0"
server_key_version: "encryptionkey@0"
I20260620 23:12:47.266719 17466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:47.267477 17481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:47.267757 17466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:47.268043 17466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "4526ace6898b43db97060eca562e3071"
format_stamp: "Formatted at 2026-06-20 23:12:47 on dist-test-slave-pswb"
server_key: "833ee37eca7fe04af30c561718ef5f31"
server_key_iv: "768c1478eb7533c853ef73b6a8bcbcf0"
server_key_version: "encryptionkey@0"
I20260620 23:12:47.268133 17466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:47.279881 17466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:47.281678 17466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:47.281827 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:47.282027 17466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:47.282311 17466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:47.282369 17466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:47.282407 17466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:47.282435 17466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:47.288350 17466 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:33677
I20260620 23:12:47.288403 17594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:33677 every 8 connection(s)
I20260620 23:12:47.288792 17466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:12:47.292992 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 17466
I20260620 23:12:47.293085 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:12:47.293301 16700 external_mini_cluster.cc:1428] Setting key a914c954e055ca60d9267c3d32c5751b
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997167, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:47.298965 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.289500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:36681 (local address 127.16.79.62:43941)
0620 23:12:47.289546 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:47.289548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:47.289866 (+ 318us) server_negotiation.cc:407] Connection header received
0620 23:12:47.290677 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:47.290678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:47.290689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:47.290749 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:47.291327 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.291848 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.293137 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.293459 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.293501 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.295917 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:47.295918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:47.295919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:47.295924 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:47.297253 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.297795 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.297796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.297796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.297851 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.298151 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.298152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.298152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.298256 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:47.298311 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:47.298820 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 23:12:47.298899 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:12:47.299088 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.289468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:43941 (local address 127.16.79.2:36681)
0620 23:12:47.289612 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:47.289652 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:47.290543 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:47.290825 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:47.290828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:47.291059 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:47.291232 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.291236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.291931 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.291931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.292985 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.292989 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.293590 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.293590 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.293726 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.294254 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:47.294265 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:47.295827 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:47.297365 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:47.297372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:47.297374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:47.297699 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:47.297932 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:47.297933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:47.297933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:47.298077 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:47.298328 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:47.298334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:47.298501 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 23:12:47.298641 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:12:47.298817 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260620 23:12:47.299641 17595 heartbeater.cc:347] Connected to a master server at 127.16.79.62:43941
I20260620 23:12:47.299736 17595 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:47.299914 17595 heartbeater.cc:511] Master 127.16.79.62:43941 requested a full tablet report, sending...
I20260620 23:12:47.300266 17270 ts_manager.cc:194] Registered new tserver with Master: 4526ace6898b43db97060eca562e3071 (127.16.79.2:33677)
I20260620 23:12:47.300650 17270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:36681
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:12:47.319396 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:43941
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:47.399266 17600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:47.399544 17600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:47.399603 17600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:47.401522 17600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:47.401576 17600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:47.401618 17600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:12:47.403679 17600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:43941
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:47.403955 17600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:47.404174 17600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:47.409354 17605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:47.409504 17600 server_base.cc:1034] running on GCE node
W20260620 23:12:47.409374 17606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:47.409423 17608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:47.409782 17600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:47.410014 17600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:47.411164 17600 hybrid_clock.cc:648] HybridClock initialized: now 1781997167411149 us; error 35 us; skew 500 ppm
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997167, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:47.414017 17600 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:12:47.414533 17600 webserver.cc:460] Webserver started at http://127.16.79.3:34121/ using document root <none> and password file <none>
I20260620 23:12:47.414826 17600 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:47.414897 17600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:47.415045 17600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:47.416127 17600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "9f4f405dec9f410db99fd7f2f9470ec9"
format_stamp: "Formatted at 2026-06-20 23:12:47 on dist-test-slave-pswb"
server_key: "a217998356ce4cf78e0596aac9a71f65"
server_key_iv: "0c25bf0905bb673cb98ec81ebda19f7a"
server_key_version: "encryptionkey@0"
I20260620 23:12:47.416502 17600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "9f4f405dec9f410db99fd7f2f9470ec9"
format_stamp: "Formatted at 2026-06-20 23:12:47 on dist-test-slave-pswb"
server_key: "a217998356ce4cf78e0596aac9a71f65"
server_key_iv: "0c25bf0905bb673cb98ec81ebda19f7a"
server_key_version: "encryptionkey@0"
I20260620 23:12:47.417879 17600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:47.418650 17615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:47.418846 17600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:12:47.418942 17600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "9f4f405dec9f410db99fd7f2f9470ec9"
format_stamp: "Formatted at 2026-06-20 23:12:47 on dist-test-slave-pswb"
server_key: "a217998356ce4cf78e0596aac9a71f65"
server_key_iv: "0c25bf0905bb673cb98ec81ebda19f7a"
server_key_version: "encryptionkey@0"
I20260620 23:12:47.419023 17600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:47.432929 17600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:47.434710 17600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:47.434859 17600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:47.435057 17600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:47.435331 17600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:47.435384 17600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:47.435422 17600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:47.435451 17600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:47.441002 17600 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:39499
I20260620 23:12:47.441048 17728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:39499 every 8 connection(s)
I20260620 23:12:47.441327 17600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:12:47.443845 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 17600
I20260620 23:12:47.443962 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:12:47.444134 16700 external_mini_cluster.cc:1428] Setting key 883db3a97ce466dda42fbc80e38d354f
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997167, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:47.450642 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.441987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:43941 (local address 127.16.79.3:58903)
0620 23:12:47.442129 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:47.442167 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:47.443021 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:47.443221 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:47.443224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:47.443448 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:47.443615 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.443618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.444577 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.444578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.445547 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.445549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.445880 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.445881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.445979 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.446355 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:47.446362 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:47.447604 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:47.449317 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:47.449322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:47.449324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:47.449544 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:47.449769 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:47.449770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:47.449770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:47.449863 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:47.450214 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:47.450219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:47.450317 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 23:12:47.450398 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:12:47.450502 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260620 23:12:47.450654 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.442027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:58903 (local address 127.16.79.62:43941)
0620 23:12:47.442072 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:47.442073 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:47.442248 (+ 175us) server_negotiation.cc:407] Connection header received
0620 23:12:47.443107 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:47.443108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:47.443117 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:47.443173 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:47.443714 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.444493 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.445618 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.445807 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.445864 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.447684 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:47.447685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:47.447685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:47.447691 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:47.449190 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.449641 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.449642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.449642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.449702 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.449966 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.449967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.449968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.450128 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:47.450213 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:47.450466 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 23:12:47.450563 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:12:47.451392 17729 heartbeater.cc:347] Connected to a master server at 127.16.79.62:43941
I20260620 23:12:47.451495 17729 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:47.451679 17729 heartbeater.cc:511] Master 127.16.79.62:43941 requested a full tablet report, sending...
I20260620 23:12:47.452171 17270 ts_manager.cc:194] Registered new tserver with Master: 9f4f405dec9f410db99fd7f2f9470ec9 (127.16.79.3:39499)
I20260620 23:12:47.452553 17270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:58903
I20260620 23:12:47.455987 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:12:47.462790 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.457501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.16.79.62:43941)
0620 23:12:47.457581 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:47.457583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:47.457640 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:12:47.457681 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:47.457681 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:47.457690 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:47.457744 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:47.458263 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.458700 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.459352 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.459517 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.459563 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.460525 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:47.460526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:47.460526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:47.460532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:47.461736 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.462057 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.462058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.462059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.462117 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.462354 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.462354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.462355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.462481 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:47.462563 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:47.462670 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 23:12:47.462730 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:12:47.464823 17270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:12:47.466646 17270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:12:47.473712 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.469393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:33877 (local address 127.0.0.1:37912)
0620 23:12:47.469733 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:47.469744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:47.469794 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:47.470118 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:47.470119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:47.470121 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:47.470187 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.470189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.471565 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.471566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.472557 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.472559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.473386 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.473387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.473475 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.473488 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:12:47.473576 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:12:47.473632 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":4,"thread_start_us":59,"threads_started":1}
I20260620 23:12:47.473723 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.469583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.16.79.1:33877)
0620 23:12:47.469952 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:47.469957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:47.469965 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:12:47.469999 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:47.470001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:47.470017 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:47.470345 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:47.470413 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.471470 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.472638 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.473298 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.473370 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.473423 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:47.473589 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 23:12:47.473626 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":76,"threads_started":1}
I20260620 23:12:47.474123 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.469585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:33677 (local address 127.0.0.1:55298)
0620 23:12:47.469778 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:47.469788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:47.469826 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:47.470165 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:47.470166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:47.470168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:47.470217 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.470218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.471330 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.471331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.472896 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.472899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.473701 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.473702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.473771 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.473775 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:12:47.474016 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 23:12:47.474045 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":32,"thread_start_us":32,"threads_started":1}
I20260620 23:12:47.474175 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.469769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.16.79.2:33677)
0620 23:12:47.469944 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:47.469948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:47.469956 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:12:47.469984 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:47.469986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:47.469997 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:47.470090 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:47.470273 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.471221 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.472995 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.473563 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.473621 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.473670 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:47.474053 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 23:12:47.474099 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260620 23:12:47.474334 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.469393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:39499 (local address 127.0.0.1:58222)
0620 23:12:47.469539 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:47.469578 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:47.469650 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:47.470095 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:47.470098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:47.470100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:47.470165 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.470167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.471450 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.471451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.472896 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.472899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.474101 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.474102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.474163 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.474167 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:12:47.474225 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 23:12:47.474264 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":33,"thread_start_us":62,"threads_started":1}
I20260620 23:12:47.474541 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.469620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.16.79.3:39499)
0620 23:12:47.469914 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:47.469919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:47.469928 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:12:47.469966 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:47.469968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:47.469984 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:47.470067 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:47.470248 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.471353 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.472986 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.473666 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.473732 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.474395 (+ 663us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:47.474418 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 23:12:47.474445 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
I20260620 23:12:47.474519 17395 tablet_service.cc:1467] Processing CreateTablet for tablet b70d8686c5c44c41acb357c8901075b1 (DEFAULT_TABLE table=test-table [id=9c86f428b5f9461986ec2704793b4c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:47.474867 17395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70d8686c5c44c41acb357c8901075b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:47.475047 17663 tablet_service.cc:1467] Processing CreateTablet for tablet b70d8686c5c44c41acb357c8901075b1 (DEFAULT_TABLE table=test-table [id=9c86f428b5f9461986ec2704793b4c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:47.475266 17663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70d8686c5c44c41acb357c8901075b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:47.477329 17529 tablet_service.cc:1467] Processing CreateTablet for tablet b70d8686c5c44c41acb357c8901075b1 (DEFAULT_TABLE table=test-table [id=9c86f428b5f9461986ec2704793b4c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:47.477635 17529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70d8686c5c44c41acb357c8901075b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:47.477823 17743 tablet_bootstrap.cc:492] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc: Bootstrap starting.
I20260620 23:12:47.477826 17744 tablet_bootstrap.cc:492] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9: Bootstrap starting.
I20260620 23:12:47.478392 17744 tablet_bootstrap.cc:654] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:47.478425 17743 tablet_bootstrap.cc:654] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:47.478878 17744 log.cc:824] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:47.478956 17743 log.cc:824] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:47.479774 17743 tablet_bootstrap.cc:492] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc: No bootstrap required, opened a new log
I20260620 23:12:47.479846 17743 ts_tablet_manager.cc:1397] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:12:47.480842 17747 tablet_bootstrap.cc:492] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071: Bootstrap starting.
I20260620 23:12:47.481451 17747 tablet_bootstrap.cc:654] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:47.481597 17744 tablet_bootstrap.cc:492] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9: No bootstrap required, opened a new log
I20260620 23:12:47.481671 17744 ts_tablet_manager.cc:1397] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 23:12:47.481953 17747 log.cc:824] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:47.481895 17743 raft_consensus.cc:348] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39499 } } peers { permanent_uuid: "a2ffacce26c14a7990933a124a985edc" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33877 } } peers { permanent_uuid: "4526ace6898b43db97060eca562e3071" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33677 } }
I20260620 23:12:47.482060 17743 raft_consensus.cc:374] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:47.482100 17743 raft_consensus.cc:729] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ffacce26c14a7990933a124a985edc, State: Initialized, Role: FOLLOWER
I20260620 23:12:47.482205 17743 consensus_queue.cc:260] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39499 } } peers { permanent_uuid: "a2ffacce26c14a7990933a124a985edc" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33877 } } peers { permanent_uuid: "4526ace6898b43db97060eca562e3071" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33677 } }
I20260620 23:12:47.482465 17743 ts_tablet_manager.cc:1428] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:12:47.482586 17461 heartbeater.cc:503] Master 127.16.79.62:43941 was elected leader, sending a full tablet report...
I20260620 23:12:47.482659 17747 tablet_bootstrap.cc:492] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071: No bootstrap required, opened a new log
I20260620 23:12:47.482722 17747 ts_tablet_manager.cc:1397] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:12:47.483394 17744 raft_consensus.cc:348] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39499 } } peers { permanent_uuid: "a2ffacce26c14a7990933a124a985edc" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33877 } } peers { permanent_uuid: "4526ace6898b43db97060eca562e3071" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33677 } }
I20260620 23:12:47.483512 17744 raft_consensus.cc:374] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:47.483533 17744 raft_consensus.cc:729] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f4f405dec9f410db99fd7f2f9470ec9, State: Initialized, Role: FOLLOWER
I20260620 23:12:47.483636 17744 consensus_queue.cc:260] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39499 } } peers { permanent_uuid: "a2ffacce26c14a7990933a124a985edc" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33877 } } peers { permanent_uuid: "4526ace6898b43db97060eca562e3071" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33677 } }
I20260620 23:12:47.483892 17744 ts_tablet_manager.cc:1428] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:12:47.483930 17729 heartbeater.cc:503] Master 127.16.79.62:43941 was elected leader, sending a full tablet report...
I20260620 23:12:47.484432 17747 raft_consensus.cc:348] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39499 } } peers { permanent_uuid: "a2ffacce26c14a7990933a124a985edc" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33877 } } peers { permanent_uuid: "4526ace6898b43db97060eca562e3071" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33677 } }
I20260620 23:12:47.484613 17747 raft_consensus.cc:374] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:47.484660 17747 raft_consensus.cc:729] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4526ace6898b43db97060eca562e3071, State: Initialized, Role: FOLLOWER
I20260620 23:12:47.484807 17747 consensus_queue.cc:260] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39499 } } peers { permanent_uuid: "a2ffacce26c14a7990933a124a985edc" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33877 } } peers { permanent_uuid: "4526ace6898b43db97060eca562e3071" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33677 } }
I20260620 23:12:47.485080 17747 ts_tablet_manager.cc:1428] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:12:47.485110 17595 heartbeater.cc:503] Master 127.16.79.62:43941 was elected leader, sending a full tablet report...
W20260620 23:12:47.539393 17596 tablet.cc:2369] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:12:47.540558 17750 raft_consensus.cc:482] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:12:47.541019 17750 raft_consensus.cc:504] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39499 } } peers { permanent_uuid: "a2ffacce26c14a7990933a124a985edc" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33877 } } peers { permanent_uuid: "4526ace6898b43db97060eca562e3071" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33677 } }
I20260620 23:12:47.541375 17750 leader_election.cc:288] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2ffacce26c14a7990933a124a985edc (127.16.79.1:33877), 4526ace6898b43db97060eca562e3071 (127.16.79.2:33677)
I20260620 23:12:47.544283 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.541621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:59905 (local address 127.16.79.1:33877)
0620 23:12:47.541666 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:47.541668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:47.541879 (+ 211us) server_negotiation.cc:407] Connection header received
0620 23:12:47.541906 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:47.541907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:47.541922 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:47.541976 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:47.542142 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.542620 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.543556 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.543897 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.543950 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.543958 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:47.544172 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 23:12:47.544213 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:12:47.544299 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.541548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:33677 (local address 127.16.79.3:39277)
0620 23:12:47.541610 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:47.541621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:47.541662 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:47.541862 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:47.541863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:47.541866 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:47.541943 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.541945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.542613 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.542613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.543371 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.543373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.544046 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.544047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.544105 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.544107 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:12:47.544184 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 23:12:47.544217 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 23:12:47.544303 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.541636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:39277 (local address 127.16.79.2:33677)
0620 23:12:47.541675 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:47.541677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:47.541685 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:12:47.541715 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:47.541716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:47.541729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:47.541804 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:47.542018 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.542528 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.543463 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.543966 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.544032 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.544041 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:47.544203 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 23:12:47.544242 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:12:47.544313 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.541548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:33877 (local address 127.16.79.3:59905)
0620 23:12:47.541754 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:47.541765 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:47.541824 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:47.542006 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:47.542007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:47.542010 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:47.542073 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.542074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.542708 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.542709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.543480 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:47.543482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.544017 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.544018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:47.544094 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.544097 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:12:47.544140 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 23:12:47.544202 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":25,"thread_start_us":101,"threads_started":1}
I20260620 23:12:47.544673 17549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70d8686c5c44c41acb357c8901075b1" candidate_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4526ace6898b43db97060eca562e3071" is_pre_election: true
I20260620 23:12:47.544822 17415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70d8686c5c44c41acb357c8901075b1" candidate_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2ffacce26c14a7990933a124a985edc" is_pre_election: true
I20260620 23:12:47.544884 17549 raft_consensus.cc:2454] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f4f405dec9f410db99fd7f2f9470ec9 in term 0.
I20260620 23:12:47.544955 17415 raft_consensus.cc:2454] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f4f405dec9f410db99fd7f2f9470ec9 in term 0.
I20260620 23:12:47.545121 17616 leader_election.cc:302] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4526ace6898b43db97060eca562e3071, 9f4f405dec9f410db99fd7f2f9470ec9; no voters:
I20260620 23:12:47.545293 17750 raft_consensus.cc:2781] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:12:47.545367 17750 raft_consensus.cc:482] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:12:47.545394 17750 raft_consensus.cc:3037] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:47.546090 17750 raft_consensus.cc:504] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39499 } } peers { permanent_uuid: "a2ffacce26c14a7990933a124a985edc" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33877 } } peers { permanent_uuid: "4526ace6898b43db97060eca562e3071" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33677 } }
I20260620 23:12:47.546276 17750 leader_election.cc:288] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [CANDIDATE]: Term 1 election: Requested vote from peers a2ffacce26c14a7990933a124a985edc (127.16.79.1:33877), 4526ace6898b43db97060eca562e3071 (127.16.79.2:33677)
I20260620 23:12:47.546343 17415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70d8686c5c44c41acb357c8901075b1" candidate_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2ffacce26c14a7990933a124a985edc"
I20260620 23:12:47.546361 17549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70d8686c5c44c41acb357c8901075b1" candidate_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4526ace6898b43db97060eca562e3071"
I20260620 23:12:47.546409 17415 raft_consensus.cc:3037] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:47.546452 17549 raft_consensus.cc:3037] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:47.547055 17415 raft_consensus.cc:2454] T b70d8686c5c44c41acb357c8901075b1 P a2ffacce26c14a7990933a124a985edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f4f405dec9f410db99fd7f2f9470ec9 in term 1.
I20260620 23:12:47.547272 17549 raft_consensus.cc:2454] T b70d8686c5c44c41acb357c8901075b1 P 4526ace6898b43db97060eca562e3071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f4f405dec9f410db99fd7f2f9470ec9 in term 1.
I20260620 23:12:47.547288 17619 leader_election.cc:302] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f4f405dec9f410db99fd7f2f9470ec9, a2ffacce26c14a7990933a124a985edc; no voters:
I20260620 23:12:47.547562 17750 raft_consensus.cc:2781] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:47.547823 17750 raft_consensus.cc:686] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [term 1 LEADER]: Becoming Leader. State: Replica: 9f4f405dec9f410db99fd7f2f9470ec9, State: Running, Role: LEADER
I20260620 23:12:47.547969 17750 consensus_queue.cc:237] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39499 } } peers { permanent_uuid: "a2ffacce26c14a7990933a124a985edc" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33877 } } peers { permanent_uuid: "4526ace6898b43db97060eca562e3071" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33677 } }
I20260620 23:12:47.549191 17270 catalog_manager.cc:5495] T b70d8686c5c44c41acb357c8901075b1 P 9f4f405dec9f410db99fd7f2f9470ec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f4f405dec9f410db99fd7f2f9470ec9 (127.16.79.3). New cstate: current_term: 1 leader_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f405dec9f410db99fd7f2f9470ec9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2ffacce26c14a7990933a124a985edc" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4526ace6898b43db97060eca562e3071" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33677 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.1@KRBTEST.COM
I20260620 23:12:47.574187 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.566120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.16.79.1:33877)
0620 23:12:47.566198 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:47.566200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:47.566263 (+ 63us) server_negotiation.cc:407] Connection header received
0620 23:12:47.566301 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:47.566302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:47.566311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:47.566366 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:47.566936 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.567392 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.568075 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.568262 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.568325 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.570772 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:47.570787 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:47.570790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:47.570797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:47.572403 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.573119 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.573122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.573122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.573194 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.573494 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.573494 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.573495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.573691 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:47.573873 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:47.574001 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 23:12:47.574085 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.79.1@KRBTEST.COM
I20260620 23:12:47.598443 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.590609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.16.79.1:33877)
0620 23:12:47.590710 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:47.590713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:47.590818 (+ 105us) server_negotiation.cc:407] Connection header received
0620 23:12:47.590910 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:47.590911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:47.590922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:47.590990 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:47.591729 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.592341 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.593238 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.593435 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.593504 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.595658 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:47.595659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:47.595659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:47.595666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:47.597286 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.597663 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.597664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.597665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.597721 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.597891 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.597891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.597892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.597979 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:47.598054 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:47.598282 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 23:12:47.598348 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.1@KRBTEST.COM
I20260620 23:12:47.621055 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:47.613042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.16.79.1:33877)
0620 23:12:47.613124 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:47.613126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:47.613231 (+ 105us) server_negotiation.cc:407] Connection header received
0620 23:12:47.613304 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:47.613305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:47.613316 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:47.613397 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:47.614003 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.614541 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.615451 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:47.615651 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:47.615728 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:47.618548 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:47.618549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:47.618550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:47.618556 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:47.619926 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.620219 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.620220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.620220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.620266 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:47.620425 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:47.620426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:47.620426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:47.620504 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:47.620572 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:47.620855 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 23:12:47.620964 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 23:12:47.621513 17375 tablet_service.cc:2908] Rejecting scan request for tablet b70d8686c5c44c41acb357c8901075b1: Uninitialized: clean time has not yet been initialized
I20260620 23:12:47.622339 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 17332
I20260620 23:12:47.627851 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 17466
I20260620 23:12:47.632145 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 17600
I20260620 23:12:47.636608 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 17238
2026-06-20T23:12:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1942 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17767](info): set up 2 sockets
Jun 20 23:12:47 dist-test-slave-pswb krb5kdc[17767](info): commencing operation
krb5kdc: starting...
W20260620 23:12:48.676332 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:12:48 dist-test-slave-pswb krb5kdc[17767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:12:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:12:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:12:48.791347 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:33041
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:33041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:48.869553 17780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:48.869742 17780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:48.869763 17780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:48.870888 17780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:12:48.870929 17780 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:12:48.870942 17780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:48.870955 17780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:48.870991 17780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:12:48.871021 17780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:12:48.872462 17780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:33041
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:33041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:48.872696 17780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:48.872964 17780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:48.877826 17785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:48.877868 17786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:48.877848 17788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:48.878161 17780 server_base.cc:1034] running on GCE node
I20260620 23:12:48.878399 17780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:48.878659 17780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:48.879823 17780 hybrid_clock.cc:648] HybridClock initialized: now 1781997168879807 us; error 25 us; skew 500 ppm
Jun 20 23:12:48 dist-test-slave-pswb krb5kdc[17767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997168, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:48.882805 17780 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:12:48.883263 17780 webserver.cc:460] Webserver started at http://127.16.79.62:45279/ using document root <none> and password file <none>
I20260620 23:12:48.883512 17780 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:48.883558 17780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:48.883697 17780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:48.884681 17780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "fa454770d5784e809fbd5bda43b5006b"
format_stamp: "Formatted at 2026-06-20 23:12:48 on dist-test-slave-pswb"
server_key: "4a3a85b542cec1425e784886d35c913e"
server_key_iv: "0e0f699538b6ab4baee60591c28b67a5"
server_key_version: "encryptionkey@0"
I20260620 23:12:48.885098 17780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "fa454770d5784e809fbd5bda43b5006b"
format_stamp: "Formatted at 2026-06-20 23:12:48 on dist-test-slave-pswb"
server_key: "4a3a85b542cec1425e784886d35c913e"
server_key_iv: "0e0f699538b6ab4baee60591c28b67a5"
server_key_version: "encryptionkey@0"
I20260620 23:12:48.886397 17780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:48.887116 17795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:48.887287 17780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:12:48.887392 17780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "fa454770d5784e809fbd5bda43b5006b"
format_stamp: "Formatted at 2026-06-20 23:12:48 on dist-test-slave-pswb"
server_key: "4a3a85b542cec1425e784886d35c913e"
server_key_iv: "0e0f699538b6ab4baee60591c28b67a5"
server_key_version: "encryptionkey@0"
I20260620 23:12:48.887475 17780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:48.895955 17780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:48.898914 17780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:48.899102 17780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:48.904026 17780 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:33041
I20260620 23:12:48.904076 17857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:33041 every 8 connection(s)
I20260620 23:12:48.904387 17780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:12:48.905129 17858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:48.905382 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 17780
I20260620 23:12:48.905483 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:12:48.905642 16700 external_mini_cluster.cc:1428] Setting key 6010af9f68e4eb68745262acf976bb14
I20260620 23:12:48.907997 17858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b: Bootstrap starting.
I20260620 23:12:48.908572 17858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:48.908876 17858 log.cc:824] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:48.909559 17858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b: No bootstrap required, opened a new log
I20260620 23:12:48.911621 17858 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa454770d5784e809fbd5bda43b5006b" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33041 } }
I20260620 23:12:48.911744 17858 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:48.911782 17858 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa454770d5784e809fbd5bda43b5006b, State: Initialized, Role: FOLLOWER
I20260620 23:12:48.911911 17858 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa454770d5784e809fbd5bda43b5006b" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33041 } }
I20260620 23:12:48.912002 17858 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:12:48.912043 17858 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:12:48.912098 17858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [term 0 FOLLOWER]: Advancing to term 1
Jun 20 23:12:48 dist-test-slave-pswb krb5kdc[17767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:48.912626 17858 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa454770d5784e809fbd5bda43b5006b" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33041 } }
I20260620 23:12:48.912768 17858 leader_election.cc:302] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa454770d5784e809fbd5bda43b5006b; no voters:
I20260620 23:12:48.912954 17858 leader_election.cc:288] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:12:48.913008 17863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:48.913164 17863 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [term 1 LEADER]: Becoming Leader. State: Replica: fa454770d5784e809fbd5bda43b5006b, State: Running, Role: LEADER
I20260620 23:12:48.913244 17858 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:12:48.913340 17863 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa454770d5784e809fbd5bda43b5006b" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33041 } }
I20260620 23:12:48.913751 17864 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa454770d5784e809fbd5bda43b5006b. Latest consensus state: current_term: 1 leader_uuid: "fa454770d5784e809fbd5bda43b5006b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa454770d5784e809fbd5bda43b5006b" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33041 } } }
I20260620 23:12:48.913919 17864 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:48.913756 17865 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa454770d5784e809fbd5bda43b5006b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa454770d5784e809fbd5bda43b5006b" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33041 } } }
I20260620 23:12:48.914175 17865 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:48.914163 17872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:12:48.914786 17872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:12:48.916414 17872 catalog_manager.cc:1347] Generated new cluster ID: b1a155bd585f4b4e8f24528fb085815a
I20260620 23:12:48.916473 17872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:12:48.917011 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:48.906434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.16.79.62:33041)
0620 23:12:48.906600 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:48.906615 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:48.906637 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:12:48.907028 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:48.907034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:48.907075 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:48.907188 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:48.908023 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:48.909083 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:48.909865 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:48.910453 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:48.910584 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:48.913067 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:48.913071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:48.913072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:48.913084 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:48.915115 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:48.915509 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:48.915512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:48.915512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:48.915614 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:48.915827 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:48.915828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:48.915828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:48.916109 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:48.916249 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:48.916660 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 23:12:48.916827 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260620 23:12:48.932555 17872 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:12:48.933169 17872 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:12:48.940301 17872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b: Generated new TSK 0
I20260620 23:12:48.940506 17872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:12:48.971648 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:33041
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:49.056505 17884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:49.056756 17884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:49.056814 17884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:49.058369 17884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:49.058436 17884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:49.058473 17884 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:12:49.058538 17884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:12:49.060027 17884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.79.62:33041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:49.060281 17884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:49.060498 17884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:49.065501 17890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:49.065488 17884 server_base.cc:1034] running on GCE node
W20260620 23:12:49.065487 17889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:49.065704 17892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:49.065963 17884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:49.066210 17884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:49.067375 17884 hybrid_clock.cc:648] HybridClock initialized: now 1781997169067358 us; error 39 us; skew 500 ppm
Jun 20 23:12:49 dist-test-slave-pswb krb5kdc[17767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997169, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:49.070058 17884 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:12:49.070551 17884 webserver.cc:460] Webserver started at http://127.16.79.1:35249/ using document root <none> and password file <none>
I20260620 23:12:49.070784 17884 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:49.070854 17884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:49.071007 17884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:49.072012 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "ef5a0e58326e4de8b74586f2ff8514c1"
format_stamp: "Formatted at 2026-06-20 23:12:49 on dist-test-slave-pswb"
server_key: "ac0dbdd197e701397345c1ec93cd2ca1"
server_key_iv: "e0d2d0f6fb6e8a006177e294f3d019c6"
server_key_version: "encryptionkey@0"
I20260620 23:12:49.072365 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "ef5a0e58326e4de8b74586f2ff8514c1"
format_stamp: "Formatted at 2026-06-20 23:12:49 on dist-test-slave-pswb"
server_key: "ac0dbdd197e701397345c1ec93cd2ca1"
server_key_iv: "e0d2d0f6fb6e8a006177e294f3d019c6"
server_key_version: "encryptionkey@0"
I20260620 23:12:49.073745 17884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:12:49.074595 17899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:49.074790 17884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:49.074894 17884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "ef5a0e58326e4de8b74586f2ff8514c1"
format_stamp: "Formatted at 2026-06-20 23:12:49 on dist-test-slave-pswb"
server_key: "ac0dbdd197e701397345c1ec93cd2ca1"
server_key_iv: "e0d2d0f6fb6e8a006177e294f3d019c6"
server_key_version: "encryptionkey@0"
I20260620 23:12:49.074976 17884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:49.091437 17884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:49.093578 17884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:49.093745 17884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:49.094272 17884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:49.094331 17884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:49.094380 17884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:49.094410 17884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:12:49 dist-test-slave-pswb krb5kdc[17767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997169, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:49.101541 17884 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:42939
I20260620 23:12:49.101840 18014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:42939 every 8 connection(s)
I20260620 23:12:49.101892 17884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:12:49.104417 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.095033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:55489 (local address 127.16.79.62:33041)
0620 23:12:49.095099 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.095102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.095538 (+ 436us) server_negotiation.cc:407] Connection header received
0620 23:12:49.096500 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.096502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.096516 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.096592 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:49.097218 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.097744 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.098725 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.098956 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.099037 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.101215 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:49.101216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:49.101217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:49.101223 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:49.102819 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.103255 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.103256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.103256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.103316 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.103801 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.103802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.103802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.103901 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:49.103961 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.104273 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.104352 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 23:12:49.104553 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.094810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33041 (local address 127.16.79.1:55489)
0620 23:12:49.095197 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.095261 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.096371 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.096672 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.096677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.096946 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:49.097118 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.097122 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.097838 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.097839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.098623 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.098625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.099053 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.099054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.099177 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.099655 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:49.099662 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:49.101129 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:49.102891 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.102895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.102897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.103111 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.103468 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.103470 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.103471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.103684 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.104001 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:49.104007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:49.104134 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.104231 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.104366 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":93,"threads_started":1}
I20260620 23:12:49.105019 18015 heartbeater.cc:347] Connected to a master server at 127.16.79.62:33041
I20260620 23:12:49.105101 18015 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:49.105289 18015 heartbeater.cc:511] Master 127.16.79.62:33041 requested a full tablet report, sending...
I20260620 23:12:49.105763 17811 ts_manager.cc:194] Registered new tserver with Master: ef5a0e58326e4de8b74586f2ff8514c1 (127.16.79.1:42939)
I20260620 23:12:49.106459 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 17884
I20260620 23:12:49.106482 17811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:55489
I20260620 23:12:49.106561 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:12:49.106748 16700 external_mini_cluster.cc:1428] Setting key 862797fbbdcd2b13596febc6b9e7068b
I20260620 23:12:49.112615 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.106112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:52683 (local address 127.16.79.62:33041)
0620 23:12:49.106146 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.106148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.106298 (+ 150us) server_negotiation.cc:407] Connection header received
0620 23:12:49.107023 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.107024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.107033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.107089 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:49.107413 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.108075 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.108849 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.109144 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.109261 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.110186 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:49.110188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:49.110188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:49.110194 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:49.111495 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.111860 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.111861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.111861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.111915 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.112124 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.112125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.112126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.112231 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:49.112305 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.112454 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.112534 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 23:12:49.112629 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.106050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33041 (local address 127.16.79.1:52683)
0620 23:12:49.106207 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.106215 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.106948 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.107095 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.107096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.107269 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:49.107323 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.107324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.108173 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.108173 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.108755 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.108757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.109264 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.109265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.109342 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.109782 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:49.109787 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:49.110077 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:49.111595 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.111596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.111596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.111776 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.111987 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.111987 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.111988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.112055 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.112298 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:49.112298 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:49.112342 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.112430 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.112528 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:12:49.132130 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:33041
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:49.215232 18024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:49.215415 18024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:49.215437 18024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:49.216547 18024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:49.216586 18024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:49.216607 18024 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:12:49.216624 18024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:12:49.218027 18024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.79.62:33041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:49.218250 18024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:49.218483 18024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:49.224155 18029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:49.224334 18024 server_base.cc:1034] running on GCE node
W20260620 23:12:49.224151 18032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:49.224152 18030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:49.224671 18024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:49.224957 18024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:49.226135 18024 hybrid_clock.cc:648] HybridClock initialized: now 1781997169226116 us; error 31 us; skew 500 ppm
Jun 20 23:12:49 dist-test-slave-pswb krb5kdc[17767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997169, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:49.229246 18024 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:12:49.229751 18024 webserver.cc:460] Webserver started at http://127.16.79.2:38751/ using document root <none> and password file <none>
I20260620 23:12:49.229988 18024 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:49.230057 18024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:49.230190 18024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:49.231173 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6"
format_stamp: "Formatted at 2026-06-20 23:12:49 on dist-test-slave-pswb"
server_key: "c1f69652d1b7b8dc41425697d9416e36"
server_key_iv: "9e6027d45e715f68e61ea8f4d4f49be8"
server_key_version: "encryptionkey@0"
I20260620 23:12:49.231527 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6"
format_stamp: "Formatted at 2026-06-20 23:12:49 on dist-test-slave-pswb"
server_key: "c1f69652d1b7b8dc41425697d9416e36"
server_key_iv: "9e6027d45e715f68e61ea8f4d4f49be8"
server_key_version: "encryptionkey@0"
I20260620 23:12:49.232889 18024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:12:49.233649 18039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:49.233817 18024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:12:49.233891 18024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6"
format_stamp: "Formatted at 2026-06-20 23:12:49 on dist-test-slave-pswb"
server_key: "c1f69652d1b7b8dc41425697d9416e36"
server_key_iv: "9e6027d45e715f68e61ea8f4d4f49be8"
server_key_version: "encryptionkey@0"
I20260620 23:12:49.233953 18024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:49.249126 18024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:49.251264 18024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:49.251370 18024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:49.251793 18024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:49.251834 18024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:49.251857 18024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:49.251894 18024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:12:49 dist-test-slave-pswb krb5kdc[17767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997169, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:49.260015 18024 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:46801
I20260620 23:12:49.260442 18024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:12:49.261725 18154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:46801 every 8 connection(s)
I20260620 23:12:49.262996 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.252309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:47367 (local address 127.16.79.62:33041)
0620 23:12:49.252380 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.252383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.252545 (+ 162us) server_negotiation.cc:407] Connection header received
0620 23:12:49.253077 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.253079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.253092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.253175 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:49.253783 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.254577 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.256263 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.256554 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.256609 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.259976 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:49.259978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:49.259978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:49.259986 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:49.261547 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.262035 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.262036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.262037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.262098 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.262372 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.262373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.262373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.262491 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:49.262559 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.262834 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.262905 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:12:49.263185 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.252240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33041 (local address 127.16.79.2:47367)
0620 23:12:49.252407 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.252462 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.252983 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.253221 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.253224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.253481 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:49.253691 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.253695 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.254678 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.254679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.256161 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.256164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.256784 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.256787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.257098 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.257847 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:49.257861 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:49.259878 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:49.261636 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.261642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.261644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.261940 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.262173 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.262174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.262174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.262296 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.262563 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:49.262576 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:49.262704 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.262803 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.262966 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260620 23:12:49.263667 18155 heartbeater.cc:347] Connected to a master server at 127.16.79.62:33041
I20260620 23:12:49.263763 18155 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:49.263957 18155 heartbeater.cc:511] Master 127.16.79.62:33041 requested a full tablet report, sending...
I20260620 23:12:49.264292 17812 ts_manager.cc:194] Registered new tserver with Master: 9dd5b291ea0e44f8ab89bcf4a8433fd6 (127.16.79.2:46801)
I20260620 23:12:49.264739 17812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:47367
I20260620 23:12:49.266640 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 18024
I20260620 23:12:49.266741 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:12:49.266920 16700 external_mini_cluster.cc:1428] Setting key ebdcbc78fb9d92f66b687cbdf36b441c
I20260620 23:12:49.271369 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.264756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:38865 (local address 127.16.79.62:33041)
0620 23:12:49.264797 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.264799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.264942 (+ 143us) server_negotiation.cc:407] Connection header received
0620 23:12:49.264970 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.264971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.264981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.265051 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:49.265401 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.265836 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.266523 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.266700 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.266812 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.267833 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:49.267834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:49.267834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:49.267840 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:49.269899 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.270372 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.270374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.270375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.270459 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.270716 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.270717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.270717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.270948 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:49.271028 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.271194 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.271272 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:12:49.271440 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.264659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33041 (local address 127.16.79.2:38865)
0620 23:12:49.264864 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.264874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.264932 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.265066 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.265067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.265265 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:49.265329 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.265332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.265919 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.265919 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.266393 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.266395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.266811 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.266812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.266893 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.267378 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:49.267383 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:49.267755 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:49.270111 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.270112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.270112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.270274 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.270566 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.270567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.270567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.270642 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.271035 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:49.271036 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:49.271084 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.271172 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.271287 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:12:49.293673 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:33041
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:49.377521 18164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:49.377704 18164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:49.377727 18164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:49.378888 18164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:49.378958 18164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:49.378993 18164 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:12:49.379033 18164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:12:49.380429 18164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.79.62:33041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:49.380683 18164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:49.380931 18164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:49.385654 18170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:49.385665 18169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:49.385830 18164 server_base.cc:1034] running on GCE node
W20260620 23:12:49.385885 18172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:49.386053 18164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:49.386253 18164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:49.387426 18164 hybrid_clock.cc:648] HybridClock initialized: now 1781997169387416 us; error 36 us; skew 500 ppm
Jun 20 23:12:49 dist-test-slave-pswb krb5kdc[17767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997169, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:49.389925 18164 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:12:49.390319 18164 webserver.cc:460] Webserver started at http://127.16.79.3:37625/ using document root <none> and password file <none>
I20260620 23:12:49.390524 18164 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:49.390588 18164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:49.390739 18164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:49.391882 18164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "d032dc9235dc427e8301469bc635adb0"
format_stamp: "Formatted at 2026-06-20 23:12:49 on dist-test-slave-pswb"
server_key: "f186682c5ba30e176fff8098d7fb66ef"
server_key_iv: "4aa48a1cc7e431f3ee3ad4259e6e9d30"
server_key_version: "encryptionkey@0"
I20260620 23:12:49.392226 18164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "d032dc9235dc427e8301469bc635adb0"
format_stamp: "Formatted at 2026-06-20 23:12:49 on dist-test-slave-pswb"
server_key: "f186682c5ba30e176fff8098d7fb66ef"
server_key_iv: "4aa48a1cc7e431f3ee3ad4259e6e9d30"
server_key_version: "encryptionkey@0"
I20260620 23:12:49.393607 18164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:12:49.394411 18179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:49.394652 18164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:49.394739 18164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "d032dc9235dc427e8301469bc635adb0"
format_stamp: "Formatted at 2026-06-20 23:12:49 on dist-test-slave-pswb"
server_key: "f186682c5ba30e176fff8098d7fb66ef"
server_key_iv: "4aa48a1cc7e431f3ee3ad4259e6e9d30"
server_key_version: "encryptionkey@0"
I20260620 23:12:49.394821 18164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:49.406294 18164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:49.408385 18164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:49.408524 18164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:49.409112 18164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:49.409150 18164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:49.409170 18164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:49.409183 18164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:12:49 dist-test-slave-pswb krb5kdc[17767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997169, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:49.415846 18164 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:34277
I20260620 23:12:49.415958 18294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:34277 every 8 connection(s)
I20260620 23:12:49.416193 18164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:12:49.418849 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.409451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:52363 (local address 127.16.79.62:33041)
0620 23:12:49.409491 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.409493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.409705 (+ 212us) server_negotiation.cc:407] Connection header received
0620 23:12:49.410173 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.410174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.410185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.410239 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:49.410866 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.411656 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.412723 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.412926 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.413016 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.415984 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:49.415985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:49.415986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:49.415993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:49.417717 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.418108 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.418109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.418109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.418163 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.418389 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.418390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.418390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.418486 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:49.418540 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.418727 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.418785 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:12:49.418970 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.409463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33041 (local address 127.16.79.3:52363)
0620 23:12:49.409593 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.409624 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.410073 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.410330 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.410333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.410608 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:49.410775 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.410779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.411746 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.411747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.412599 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.412601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.413027 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.413028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.413210 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.413792 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:49.413801 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:49.415879 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:49.417790 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.417795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.417796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.417995 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.418246 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.418246 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.418247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.418333 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.418534 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:49.418536 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:49.418628 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.418698 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.418803 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":77,"threads_started":1}
I20260620 23:12:49.419433 18295 heartbeater.cc:347] Connected to a master server at 127.16.79.62:33041
I20260620 23:12:49.419492 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 18164
I20260620 23:12:49.419512 18295 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:49.419577 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:12:49.419688 18295 heartbeater.cc:511] Master 127.16.79.62:33041 requested a full tablet report, sending...
I20260620 23:12:49.419749 16700 external_mini_cluster.cc:1428] Setting key dbac42067189243d45d5aab2fdd14cc5
I20260620 23:12:49.419993 17812 ts_manager.cc:194] Registered new tserver with Master: d032dc9235dc427e8301469bc635adb0 (127.16.79.3:34277)
I20260620 23:12:49.420374 17812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:52363
I20260620 23:12:49.421353 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:12:49.427928 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.420747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:53191 (local address 127.16.79.62:33041)
0620 23:12:49.420786 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.420788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.421464 (+ 676us) server_negotiation.cc:407] Connection header received
0620 23:12:49.421538 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.421539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.421556 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.421607 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:49.421882 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.422559 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.423800 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.423969 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.424027 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.425279 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:49.425280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:49.425280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:49.425286 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:49.426618 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.427082 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.427083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.427083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.427136 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.427417 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.427418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.427418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.427518 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:49.427594 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.427773 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.427847 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:12:49.427984 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.421262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33041 (local address 127.16.79.3:53191)
0620 23:12:49.421383 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.421393 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.421477 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.421622 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.421622 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.421759 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:49.421814 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.421815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.422730 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.422732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.423664 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.423668 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.424057 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.424058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.424149 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.424674 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:49.424680 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:49.425187 (+ 507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:49.426709 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.426712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.426717 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.426988 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.427244 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.427246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.427246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.427318 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.427597 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:49.427598 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:49.427659 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.427746 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.427875 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
Jun 20 23:12:49 dist-test-slave-pswb krb5kdc[17767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997169, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:12:49 dist-test-slave-pswb krb5kdc[17767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997169, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:49.446051 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.439484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.16.79.62:33041)
0620 23:12:49.439536 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.439537 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.439561 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:12:49.439577 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.439578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.439585 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.439626 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:49.440132 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.440620 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.441408 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.441578 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.441634 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.443785 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:49.443787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:49.443787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:49.443793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:49.445163 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.445469 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.445470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.445470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.445516 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.445671 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.445671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.445671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.445764 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:49.445833 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.445919 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.445968 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:12:49.447695 17811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:12:49.449304 17811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:12:49.456213 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.451593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:46801 (local address 127.0.0.1:53320)
0620 23:12:49.451970 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.452004 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.452069 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.452773 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.452774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.452777 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:49.452845 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.452847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.454145 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.454146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.455175 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.455177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.455901 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.455902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.455977 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.455989 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.456085 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.456137 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":85,"threads_started":1}
I20260620 23:12:49.456367 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.452040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:42939 (local address 127.0.0.1:45838)
0620 23:12:49.452355 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.452366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.452420 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.452657 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.452661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.452666 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:49.452895 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.452899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.453858 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.453859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.454995 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.454998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.456022 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.456023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.456085 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.456089 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.456139 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.456191 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":85,"threads_started":1}
I20260620 23:12:49.456496 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.452146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.16.79.1:42939)
0620 23:12:49.452322 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.452326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.452480 (+ 154us) server_negotiation.cc:407] Connection header received
0620 23:12:49.452513 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.452514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.452548 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.452630 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:49.453012 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.453767 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.455082 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.455861 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.455939 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.455987 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.456256 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.456401 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":118,"threads_started":1}
I20260620 23:12:49.456241 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.451797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.16.79.2:46801)
0620 23:12:49.452241 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.452245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.452339 (+ 94us) server_negotiation.cc:407] Connection header received
0620 23:12:49.452436 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.452439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.452462 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.452567 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:49.452948 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.454030 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.455256 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.455801 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.455899 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.455956 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.456109 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.456148 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":100,"threads_started":1}
I20260620 23:12:49.456945 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.451802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.16.79.3:34277)
0620 23:12:49.451947 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.451951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.452652 (+ 701us) server_negotiation.cc:407] Connection header received
0620 23:12:49.452705 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.452708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.452794 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.452992 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:49.453222 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.454218 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.455456 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.456259 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.456323 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.456374 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.456808 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.456845 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260620 23:12:49.457149 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.451692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:34277 (local address 127.0.0.1:40678)
0620 23:12:49.452010 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.452025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.452107 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.453078 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.453079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.453082 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:49.453136 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.453137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.454300 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.454301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.455374 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.455376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.456441 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.456442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.456501 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.456505 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.456554 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.457070 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":3,"thread_start_us":198,"threads_started":1}
I20260620 23:12:49.457214 18089 tablet_service.cc:1467] Processing CreateTablet for tablet 753712ebc7ae449b9326585151744df6 (DEFAULT_TABLE table=test-table [id=9709ecee664845929b3a34aa09bb7a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:49.457214 17949 tablet_service.cc:1467] Processing CreateTablet for tablet 753712ebc7ae449b9326585151744df6 (DEFAULT_TABLE table=test-table [id=9709ecee664845929b3a34aa09bb7a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:49.457545 18089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753712ebc7ae449b9326585151744df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:49.457545 17949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753712ebc7ae449b9326585151744df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:49.457719 18229 tablet_service.cc:1467] Processing CreateTablet for tablet 753712ebc7ae449b9326585151744df6 (DEFAULT_TABLE table=test-table [id=9709ecee664845929b3a34aa09bb7a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:49.457942 18229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753712ebc7ae449b9326585151744df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:49.460201 18318 tablet_bootstrap.cc:492] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0: Bootstrap starting.
I20260620 23:12:49.461091 18318 tablet_bootstrap.cc:654] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:49.461223 18319 tablet_bootstrap.cc:492] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1: Bootstrap starting.
I20260620 23:12:49.461438 18318 log.cc:824] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:49.461817 18319 tablet_bootstrap.cc:654] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:49.462132 18319 log.cc:824] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:49.462227 18318 tablet_bootstrap.cc:492] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0: No bootstrap required, opened a new log
I20260620 23:12:49.462307 18318 ts_tablet_manager.cc:1397] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:12:49.462430 18320 tablet_bootstrap.cc:492] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Bootstrap starting.
I20260620 23:12:49.462931 18320 tablet_bootstrap.cc:654] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:49.463219 18320 log.cc:824] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:49.463884 18320 tablet_bootstrap.cc:492] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: No bootstrap required, opened a new log
I20260620 23:12:49.463948 18320 ts_tablet_manager.cc:1397] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:12:49.464082 18318 raft_consensus.cc:348] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } }
I20260620 23:12:49.464224 18318 raft_consensus.cc:374] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:49.464249 18318 raft_consensus.cc:729] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d032dc9235dc427e8301469bc635adb0, State: Initialized, Role: FOLLOWER
I20260620 23:12:49.464377 18318 consensus_queue.cc:260] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } }
I20260620 23:12:49.464670 18318 ts_tablet_manager.cc:1428] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 23:12:49.464727 18295 heartbeater.cc:503] Master 127.16.79.62:33041 was elected leader, sending a full tablet report...
I20260620 23:12:49.466068 18320 raft_consensus.cc:348] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } }
I20260620 23:12:49.466212 18320 raft_consensus.cc:374] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:49.466255 18320 raft_consensus.cc:729] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd5b291ea0e44f8ab89bcf4a8433fd6, State: Initialized, Role: FOLLOWER
I20260620 23:12:49.466361 18320 consensus_queue.cc:260] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } }
I20260620 23:12:49.466607 18320 ts_tablet_manager.cc:1428] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:12:49.466696 18155 heartbeater.cc:503] Master 127.16.79.62:33041 was elected leader, sending a full tablet report...
I20260620 23:12:49.467087 18319 tablet_bootstrap.cc:492] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1: No bootstrap required, opened a new log
I20260620 23:12:49.467177 18319 ts_tablet_manager.cc:1397] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 23:12:49.468848 18319 raft_consensus.cc:348] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } }
I20260620 23:12:49.468983 18319 raft_consensus.cc:374] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:49.469026 18319 raft_consensus.cc:729] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5a0e58326e4de8b74586f2ff8514c1, State: Initialized, Role: FOLLOWER
I20260620 23:12:49.469127 18319 consensus_queue.cc:260] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } }
I20260620 23:12:49.469376 18319 ts_tablet_manager.cc:1428] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 23:12:49.469478 18015 heartbeater.cc:503] Master 127.16.79.62:33041 was elected leader, sending a full tablet report...
W20260620 23:12:49.511431 18156 tablet.cc:2369] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:12:49.602612 18016 tablet.cc:2369] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:12:49.666765 18296 tablet.cc:2369] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:12:49.718462 18326 raft_consensus.cc:482] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:12:49.718593 18326 raft_consensus.cc:504] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } }
I20260620 23:12:49.719009 18326 leader_election.cc:288] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d032dc9235dc427e8301469bc635adb0 (127.16.79.3:34277), 9dd5b291ea0e44f8ab89bcf4a8433fd6 (127.16.79.2:46801)
I20260620 23:12:49.720290 18325 raft_consensus.cc:482] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:12:49.720431 18325 raft_consensus.cc:504] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } }
I20260620 23:12:49.720760 18325 leader_election.cc:288] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d032dc9235dc427e8301469bc635adb0 (127.16.79.3:34277), ef5a0e58326e4de8b74586f2ff8514c1 (127.16.79.1:42939)
I20260620 23:12:49.723520 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.719346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:37949 (local address 127.16.79.2:46801)
0620 23:12:49.719392 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.719394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.719447 (+ 53us) server_negotiation.cc:407] Connection header received
0620 23:12:49.719594 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.719597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.719614 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.719687 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:49.720112 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.720959 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.722584 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.723151 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.723207 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.723226 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.723425 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.723458 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":24}
I20260620 23:12:49.723521 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.719446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:59757 (local address 127.16.79.3:34277)
0620 23:12:49.719509 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.719512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.719796 (+ 284us) server_negotiation.cc:407] Connection header received
0620 23:12:49.719825 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.719826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.719839 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.719916 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:49.720380 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.721080 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.722584 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.723142 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.723189 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.723200 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.723425 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.723458 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:12:49.723788 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.719356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:34277 (local address 127.16.79.1:59757)
0620 23:12:49.719551 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.719565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.719645 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.720027 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.720029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.720193 (+ 164us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:49.720264 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.720266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.721168 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.721168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.722479 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.722482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.723256 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.723257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.723339 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.723345 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.723588 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.723641 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":49,"thread_start_us":76,"threads_started":1}
I20260620 23:12:49.724052 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.719207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:46801 (local address 127.16.79.1:37949)
0620 23:12:49.719369 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.719384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.719476 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.719822 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.719824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.719828 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:49.719944 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.719946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.721205 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.721206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.722487 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.722488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.723257 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.723257 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.723339 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.723345 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.723937 (+ 592us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.723978 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":105,"thread_start_us":79,"threads_started":1}
I20260620 23:12:49.724423 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753712ebc7ae449b9326585151744df6" candidate_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d032dc9235dc427e8301469bc635adb0" is_pre_election: true
I20260620 23:12:49.724512 18109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753712ebc7ae449b9326585151744df6" candidate_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" is_pre_election: true
I20260620 23:12:49.724579 18248 raft_consensus.cc:2454] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5a0e58326e4de8b74586f2ff8514c1 in term 0.
I20260620 23:12:49.724588 18109 raft_consensus.cc:2454] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5a0e58326e4de8b74586f2ff8514c1 in term 0.
I20260620 23:12:49.724823 17902 leader_election.cc:302] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd5b291ea0e44f8ab89bcf4a8433fd6, ef5a0e58326e4de8b74586f2ff8514c1; no voters:
I20260620 23:12:49.724958 18326 raft_consensus.cc:2781] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:12:49.725016 18326 raft_consensus.cc:482] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:12:49.725039 18326 raft_consensus.cc:3037] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:49.725131 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.721191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:53279 (local address 127.16.79.1:42939)
0620 23:12:49.721233 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.721235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.721243 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:12:49.721262 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.721263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.721270 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.721322 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:49.721519 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.722235 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.723758 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.724258 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.724300 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.724307 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.725033 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.725069 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":23}
I20260620 23:12:49.725225 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.720897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:34277 (local address 127.16.79.2:45539)
0620 23:12:49.720948 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.720959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.721002 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.721519 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.721520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.721521 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:49.721561 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.721562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.722424 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.722424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.723584 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.723587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.724950 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.724951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.725033 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.725036 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.725091 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.725137 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":65}
I20260620 23:12:49.725412 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.721086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:45539 (local address 127.16.79.3:34277)
0620 23:12:49.721383 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.721385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.721393 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:12:49.721415 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.721416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.721424 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.721617 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:49.721646 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.722332 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.723679 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.724840 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.724900 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.724909 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.725301 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.725334 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":85,"threads_started":1}
I20260620 23:12:49.725422 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.720980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:42939 (local address 127.16.79.2:53279)
0620 23:12:49.721149 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.721160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.721202 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.721336 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.721337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.721338 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:49.721431 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.721432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.722320 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.722321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.723585 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.723587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.724836 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.724837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.724926 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.724929 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.725245 (+ 316us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.725301 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":48,"thread_start_us":114,"threads_started":1}
I20260620 23:12:49.725721 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753712ebc7ae449b9326585151744df6" candidate_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d032dc9235dc427e8301469bc635adb0" is_pre_election: true
I20260620 23:12:49.725798 18248 raft_consensus.cc:2454] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd5b291ea0e44f8ab89bcf4a8433fd6 in term 0.
I20260620 23:12:49.725950 18040 leader_election.cc:302] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd5b291ea0e44f8ab89bcf4a8433fd6, d032dc9235dc427e8301469bc635adb0; no voters:
I20260620 23:12:49.726038 18326 raft_consensus.cc:504] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } }
I20260620 23:12:49.726320 18326 leader_election.cc:288] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [CANDIDATE]: Term 1 election: Requested vote from peers d032dc9235dc427e8301469bc635adb0 (127.16.79.3:34277), 9dd5b291ea0e44f8ab89bcf4a8433fd6 (127.16.79.2:46801)
I20260620 23:12:49.726311 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753712ebc7ae449b9326585151744df6" candidate_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d032dc9235dc427e8301469bc635adb0"
I20260620 23:12:49.726372 18248 raft_consensus.cc:3037] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:49.726365 18109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753712ebc7ae449b9326585151744df6" candidate_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6"
I20260620 23:12:49.726374 17969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753712ebc7ae449b9326585151744df6" candidate_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" is_pre_election: true
I20260620 23:12:49.726564 18109 raft_consensus.cc:3037] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:49.726601 17969 raft_consensus.cc:2379] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dd5b291ea0e44f8ab89bcf4a8433fd6 in current term 1: Already voted for candidate ef5a0e58326e4de8b74586f2ff8514c1 in this term.
I20260620 23:12:49.727355 18109 raft_consensus.cc:2454] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5a0e58326e4de8b74586f2ff8514c1 in term 1.
I20260620 23:12:49.727439 18325 raft_consensus.cc:2741] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 23:12:49.727468 18248 raft_consensus.cc:2454] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5a0e58326e4de8b74586f2ff8514c1 in term 1.
I20260620 23:12:49.727573 17902 leader_election.cc:302] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd5b291ea0e44f8ab89bcf4a8433fd6, ef5a0e58326e4de8b74586f2ff8514c1; no voters:
I20260620 23:12:49.727654 18326 raft_consensus.cc:2781] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:49.727777 18326 raft_consensus.cc:686] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 LEADER]: Becoming Leader. State: Replica: ef5a0e58326e4de8b74586f2ff8514c1, State: Running, Role: LEADER
I20260620 23:12:49.727854 18326 consensus_queue.cc:237] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } }
I20260620 23:12:49.729104 17811 catalog_manager.cc:5495] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5a0e58326e4de8b74586f2ff8514c1 (127.16.79.1). New cstate: current_term: 1 leader_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20260620 23:12:49.740797 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.738212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.16.79.1:42939)
0620 23:12:49.738300 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.738302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.738395 (+ 93us) server_negotiation.cc:407] Connection header received
0620 23:12:49.738506 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.738507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.738524 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.738601 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:49.738774 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.739208 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.740000 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.740181 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.740254 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.740455 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:49.740544 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:49.740629 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.740674 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.740718 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:12:49.742285 18109 raft_consensus.cc:1261] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 1 FOLLOWER]: Refusing update from remote peer ef5a0e58326e4de8b74586f2ff8514c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:49.742353 18248 raft_consensus.cc:1261] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0 [term 1 FOLLOWER]: Refusing update from remote peer ef5a0e58326e4de8b74586f2ff8514c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:49.742532 18326 consensus_queue.cc:1035] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:49.742887 18331 consensus_queue.cc:1035] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:49.746542 18336 mvcc.cc:205] Tried to move back new op lower bound from 7299060407262318592 to 7299060407205797888. Current Snapshot: MvccSnapshot[applied={T|T < 7299060407262318592 or (T in {7299060407262318592})}]
I20260620 23:12:49.748238 17812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:49.748322 17812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:49.749004 17812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b: Sending DeleteTablet for 3 replicas of tablet 753712ebc7ae449b9326585151744df6
I20260620 23:12:49.749274 18229 tablet_service.cc:1514] Processing DeleteTablet for tablet 753712ebc7ae449b9326585151744df6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:49 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:40678
I20260620 23:12:49.749392 17949 tablet_service.cc:1514] Processing DeleteTablet for tablet 753712ebc7ae449b9326585151744df6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:49 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:45838
I20260620 23:12:49.749470 18345 tablet_replica.cc:330] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0: stopping tablet replica
I20260620 23:12:49.749554 18345 raft_consensus.cc:2229] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:49.749594 18089 tablet_service.cc:1514] Processing DeleteTablet for tablet 753712ebc7ae449b9326585151744df6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:49 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:53320
I20260620 23:12:49.749650 18345 raft_consensus.cc:2258] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:49.749722 17812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:12:49.749876 17812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:12:49.750043 18346 tablet_replica.cc:330] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1: stopping tablet replica
I20260620 23:12:49.750187 18347 tablet_replica.cc:330] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: stopping tablet replica
I20260620 23:12:49.750180 18345 ts_tablet_manager.cc:1905] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:49.750327 18347 raft_consensus.cc:2229] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:49.750442 18346 raft_consensus.cc:2229] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:12:49.750573 18346 raft_consensus.cc:2258] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:49.750712 18347 raft_consensus.cc:2258] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:49.751106 17949 tablet_service.cc:1467] Processing CreateTablet for tablet c10ce164f70140adbbdd86475bb1e18a (DEFAULT_TABLE table=test-table [id=021e3bb95d94426fb59e47905efa4678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:49.751206 18346 ts_tablet_manager.cc:1905] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:49.751236 18347 ts_tablet_manager.cc:1905] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:49.751214 17949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10ce164f70140adbbdd86475bb1e18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:49.751955 18345 ts_tablet_manager.cc:1918] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:12:49.752017 18345 log.cc:1196] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/753712ebc7ae449b9326585151744df6
I20260620 23:12:49.752286 18345 ts_tablet_manager.cc:1939] T 753712ebc7ae449b9326585151744df6 P d032dc9235dc427e8301469bc635adb0: Deleting consensus metadata
I20260620 23:12:49.752841 18347 ts_tablet_manager.cc:1918] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:12:49.752888 18319 tablet_bootstrap.cc:492] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1: Bootstrap starting.
I20260620 23:12:49.752903 18347 log.cc:1196] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/753712ebc7ae449b9326585151744df6
I20260620 23:12:49.752954 18346 ts_tablet_manager.cc:1918] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:12:49.753010 18346 log.cc:1196] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/wal/wals/753712ebc7ae449b9326585151744df6
I20260620 23:12:49.753150 18347 ts_tablet_manager.cc:1939] T 753712ebc7ae449b9326585151744df6 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Deleting consensus metadata
I20260620 23:12:49.753173 17796 catalog_manager.cc:4841] TS d032dc9235dc427e8301469bc635adb0 (127.16.79.3:34277): tablet 753712ebc7ae449b9326585151744df6 (table test-table [id=9709ecee664845929b3a34aa09bb7a2f]) successfully deleted
I20260620 23:12:49.753239 18346 ts_tablet_manager.cc:1939] T 753712ebc7ae449b9326585151744df6 P ef5a0e58326e4de8b74586f2ff8514c1: Deleting consensus metadata
I20260620 23:12:49.753293 18229 tablet_service.cc:1467] Processing CreateTablet for tablet c10ce164f70140adbbdd86475bb1e18a (DEFAULT_TABLE table=test-table [id=021e3bb95d94426fb59e47905efa4678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:49.753355 18319 tablet_bootstrap.cc:654] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:49.753382 18229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10ce164f70140adbbdd86475bb1e18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:49.753997 18319 tablet_bootstrap.cc:492] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1: No bootstrap required, opened a new log
I20260620 23:12:49.754050 18319 ts_tablet_manager.cc:1397] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:49.754200 18319 raft_consensus.cc:348] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.754269 18319 raft_consensus.cc:374] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:49.754288 18319 raft_consensus.cc:729] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5a0e58326e4de8b74586f2ff8514c1, State: Initialized, Role: FOLLOWER
I20260620 23:12:49.754333 18319 consensus_queue.cc:260] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.754413 18319 ts_tablet_manager.cc:1428] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:49.754621 18318 tablet_bootstrap.cc:492] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0: Bootstrap starting.
I20260620 23:12:49.754699 17796 catalog_manager.cc:4841] TS ef5a0e58326e4de8b74586f2ff8514c1 (127.16.79.1:42939): tablet 753712ebc7ae449b9326585151744df6 (table test-table [id=9709ecee664845929b3a34aa09bb7a2f]) successfully deleted
I20260620 23:12:49.754827 17798 catalog_manager.cc:4841] TS 9dd5b291ea0e44f8ab89bcf4a8433fd6 (127.16.79.2:46801): tablet 753712ebc7ae449b9326585151744df6 (table test-table [id=9709ecee664845929b3a34aa09bb7a2f]) successfully deleted
I20260620 23:12:49.754895 18089 tablet_service.cc:1467] Processing CreateTablet for tablet c10ce164f70140adbbdd86475bb1e18a (DEFAULT_TABLE table=test-table [id=021e3bb95d94426fb59e47905efa4678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:49.754990 18089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10ce164f70140adbbdd86475bb1e18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:49.754993 18318 tablet_bootstrap.cc:654] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:49.755867 18318 tablet_bootstrap.cc:492] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0: No bootstrap required, opened a new log
I20260620 23:12:49.755916 18318 ts_tablet_manager.cc:1397] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:49.756050 18318 raft_consensus.cc:348] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.756105 18318 raft_consensus.cc:374] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:49.756125 18318 raft_consensus.cc:729] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d032dc9235dc427e8301469bc635adb0, State: Initialized, Role: FOLLOWER
I20260620 23:12:49.756168 18318 consensus_queue.cc:260] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.756251 18318 ts_tablet_manager.cc:1428] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:49.756337 18320 tablet_bootstrap.cc:492] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Bootstrap starting.
I20260620 23:12:49.756745 18320 tablet_bootstrap.cc:654] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:49.757469 18320 tablet_bootstrap.cc:492] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6: No bootstrap required, opened a new log
I20260620 23:12:49.757524 18320 ts_tablet_manager.cc:1397] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:49.757638 18320 raft_consensus.cc:348] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.757715 18320 raft_consensus.cc:374] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:49.757736 18320 raft_consensus.cc:729] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd5b291ea0e44f8ab89bcf4a8433fd6, State: Initialized, Role: FOLLOWER
I20260620 23:12:49.757776 18320 consensus_queue.cc:260] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.757854 18320 ts_tablet_manager.cc:1428] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 23:12:49.895216 18341 raft_consensus.cc:482] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:12:49.895323 18341 raft_consensus.cc:504] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.895490 18341 leader_election.cc:288] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd5b291ea0e44f8ab89bcf4a8433fd6 (127.16.79.2:46801), d032dc9235dc427e8301469bc635adb0 (127.16.79.3:34277)
I20260620 23:12:49.895704 18109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10ce164f70140adbbdd86475bb1e18a" candidate_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" is_pre_election: true
I20260620 23:12:49.895804 18109 raft_consensus.cc:2454] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5a0e58326e4de8b74586f2ff8514c1 in term 0.
I20260620 23:12:49.895968 17902 leader_election.cc:302] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd5b291ea0e44f8ab89bcf4a8433fd6, ef5a0e58326e4de8b74586f2ff8514c1; no voters:
I20260620 23:12:49.895931 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10ce164f70140adbbdd86475bb1e18a" candidate_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d032dc9235dc427e8301469bc635adb0" is_pre_election: true
I20260620 23:12:49.896083 18248 raft_consensus.cc:2454] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5a0e58326e4de8b74586f2ff8514c1 in term 0.
I20260620 23:12:49.896111 18341 raft_consensus.cc:2781] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:12:49.896159 18341 raft_consensus.cc:482] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:12:49.896193 18341 raft_consensus.cc:3037] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:49.897105 18341 raft_consensus.cc:504] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.897250 18341 leader_election.cc:288] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd5b291ea0e44f8ab89bcf4a8433fd6 (127.16.79.2:46801), d032dc9235dc427e8301469bc635adb0 (127.16.79.3:34277)
I20260620 23:12:49.897419 18109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10ce164f70140adbbdd86475bb1e18a" candidate_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6"
I20260620 23:12:49.897504 18109 raft_consensus.cc:3037] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:49.897614 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10ce164f70140adbbdd86475bb1e18a" candidate_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d032dc9235dc427e8301469bc635adb0"
I20260620 23:12:49.897761 18248 raft_consensus.cc:3037] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:49.898231 18109 raft_consensus.cc:2454] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5a0e58326e4de8b74586f2ff8514c1 in term 1.
I20260620 23:12:49.898568 17902 leader_election.cc:302] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd5b291ea0e44f8ab89bcf4a8433fd6, ef5a0e58326e4de8b74586f2ff8514c1; no voters:
I20260620 23:12:49.898694 18341 raft_consensus.cc:2781] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:49.898772 18341 raft_consensus.cc:686] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 LEADER]: Becoming Leader. State: Replica: ef5a0e58326e4de8b74586f2ff8514c1, State: Running, Role: LEADER
I20260620 23:12:49.898847 18341 consensus_queue.cc:237] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.899040 18248 raft_consensus.cc:2454] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5a0e58326e4de8b74586f2ff8514c1 in term 1.
I20260620 23:12:49.899432 17812 catalog_manager.cc:5495] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5a0e58326e4de8b74586f2ff8514c1 (127.16.79.1). New cstate: current_term: 1 leader_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:12:49 dist-test-slave-pswb krb5kdc[17767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997168, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:49.925211 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.917808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.16.79.62:33041)
0620 23:12:49.917891 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:49.917893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:49.918168 (+ 275us) server_negotiation.cc:407] Connection header received
0620 23:12:49.918194 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:49.918195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:49.918205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:49.918273 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:49.918752 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.919253 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.920281 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.920489 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.920556 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.922711 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:49.922712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:49.922712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:49.922718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:49.923883 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.924226 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.924227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.924227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.924285 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:49.924471 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:49.924472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:49.924473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:49.924585 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:49.924647 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:49.925041 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 23:12:49.925135 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":51}
I20260620 23:12:49.925212 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:49.917737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33041 (local address 127.0.0.1:56252)
0620 23:12:49.918057 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:49.918070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:49.918143 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:49.918327 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:49.918328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:49.918612 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:49.918654 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.918656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.919385 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.919386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.920194 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:49.920196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:49.920617 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:49.920618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:49.920672 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:49.921303 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:49.921307 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:49.922632 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:49.923955 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.923962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.923963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.924158 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.924347 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:49.924347 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:49.924348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:49.924400 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:49.924789 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:49.924808 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:49.924919 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 23:12:49.924989 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 23:12:49.925078 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":114,"threads_started":1}
I20260620 23:12:49.927260 17812 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:56252:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:12:49.927484 17812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:12:49.929335 17949 tablet_service.cc:1467] Processing CreateTablet for tablet 96fa6bce648745a9b8c764a3fc11dc52 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ebb2bcd3f6a44e1d9f7ffcf62890a0d3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:12:49.929445 18089 tablet_service.cc:1467] Processing CreateTablet for tablet 96fa6bce648745a9b8c764a3fc11dc52 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ebb2bcd3f6a44e1d9f7ffcf62890a0d3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:12:49.929494 17949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fa6bce648745a9b8c764a3fc11dc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:49.929481 18229 tablet_service.cc:1467] Processing CreateTablet for tablet 96fa6bce648745a9b8c764a3fc11dc52 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ebb2bcd3f6a44e1d9f7ffcf62890a0d3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:12:49.929567 18089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fa6bce648745a9b8c764a3fc11dc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:49.929687 18229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fa6bce648745a9b8c764a3fc11dc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:49.930980 18319 tablet_bootstrap.cc:492] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1: Bootstrap starting.
I20260620 23:12:49.930998 18320 tablet_bootstrap.cc:492] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Bootstrap starting.
I20260620 23:12:49.931216 18318 tablet_bootstrap.cc:492] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0: Bootstrap starting.
I20260620 23:12:49.931279 18319 tablet_bootstrap.cc:654] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:49.931396 18320 tablet_bootstrap.cc:654] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:49.931573 18318 tablet_bootstrap.cc:654] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:49.931829 18319 tablet_bootstrap.cc:492] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1: No bootstrap required, opened a new log
I20260620 23:12:49.931875 18319 ts_tablet_manager.cc:1397] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:49.932006 18320 tablet_bootstrap.cc:492] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: No bootstrap required, opened a new log
I20260620 23:12:49.931989 18319 raft_consensus.cc:348] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.932040 18320 ts_tablet_manager.cc:1397] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:49.932052 18319 raft_consensus.cc:374] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:49.932075 18319 raft_consensus.cc:729] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5a0e58326e4de8b74586f2ff8514c1, State: Initialized, Role: FOLLOWER
I20260620 23:12:49.932123 18319 consensus_queue.cc:260] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.932174 18318 tablet_bootstrap.cc:492] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0: No bootstrap required, opened a new log
I20260620 23:12:49.932173 18320 raft_consensus.cc:348] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.932214 18319 ts_tablet_manager.cc:1428] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:49.932240 18318 ts_tablet_manager.cc:1397] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:49.932235 18320 raft_consensus.cc:374] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:49.932271 18320 raft_consensus.cc:729] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd5b291ea0e44f8ab89bcf4a8433fd6, State: Initialized, Role: FOLLOWER
I20260620 23:12:49.932252 18341 tablet_replica.cc:439] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } }
I20260620 23:12:49.932336 18341 tablet_replica.cc:442] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:49.932317 18320 consensus_queue.cc:260] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.932374 18318 raft_consensus.cc:348] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.932411 18320 ts_tablet_manager.cc:1428] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:49.932441 18318 raft_consensus.cc:374] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:49.932464 18318 raft_consensus.cc:729] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d032dc9235dc427e8301469bc635adb0, State: Initialized, Role: FOLLOWER
I20260620 23:12:49.932441 18325 tablet_replica.cc:439] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } }
I20260620 23:12:49.932513 18325 tablet_replica.cc:442] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:49.932507 18318 consensus_queue.cc:260] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:49.932607 18318 ts_tablet_manager.cc:1428] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:49.932634 18324 tablet_replica.cc:439] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } }
I20260620 23:12:49.932751 18324 tablet_replica.cc:442] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:50.095402 18325 raft_consensus.cc:482] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:12:50.095506 18325 raft_consensus.cc:504] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:50.095980 18325 leader_election.cc:288] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5a0e58326e4de8b74586f2ff8514c1 (127.16.79.1:42939), d032dc9235dc427e8301469bc635adb0 (127.16.79.3:34277)
I20260620 23:12:50.096294 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fa6bce648745a9b8c764a3fc11dc52" candidate_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d032dc9235dc427e8301469bc635adb0" is_pre_election: true
I20260620 23:12:50.096352 17969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fa6bce648745a9b8c764a3fc11dc52" candidate_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" is_pre_election: true
I20260620 23:12:50.096429 18248 raft_consensus.cc:2454] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd5b291ea0e44f8ab89bcf4a8433fd6 in term 0.
I20260620 23:12:50.096488 17969 raft_consensus.cc:2454] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd5b291ea0e44f8ab89bcf4a8433fd6 in term 0.
I20260620 23:12:50.096565 18040 leader_election.cc:302] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd5b291ea0e44f8ab89bcf4a8433fd6, d032dc9235dc427e8301469bc635adb0; no voters:
I20260620 23:12:50.096662 18325 raft_consensus.cc:2781] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:12:50.096913 18325 raft_consensus.cc:482] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:12:50.096956 18325 raft_consensus.cc:3037] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:50.098013 18325 raft_consensus.cc:504] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:50.098155 18325 leader_election.cc:288] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [CANDIDATE]: Term 1 election: Requested vote from peers ef5a0e58326e4de8b74586f2ff8514c1 (127.16.79.1:42939), d032dc9235dc427e8301469bc635adb0 (127.16.79.3:34277)
I20260620 23:12:50.098446 17969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fa6bce648745a9b8c764a3fc11dc52" candidate_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5a0e58326e4de8b74586f2ff8514c1"
I20260620 23:12:50.098466 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fa6bce648745a9b8c764a3fc11dc52" candidate_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d032dc9235dc427e8301469bc635adb0"
I20260620 23:12:50.098546 18248 raft_consensus.cc:3037] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:50.098547 17969 raft_consensus.cc:3037] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:50.099242 18248 raft_consensus.cc:2454] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd5b291ea0e44f8ab89bcf4a8433fd6 in term 1.
I20260620 23:12:50.099349 17969 raft_consensus.cc:2454] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd5b291ea0e44f8ab89bcf4a8433fd6 in term 1.
I20260620 23:12:50.099392 18040 leader_election.cc:302] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd5b291ea0e44f8ab89bcf4a8433fd6, d032dc9235dc427e8301469bc635adb0; no voters:
I20260620 23:12:50.099545 18325 raft_consensus.cc:2781] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:50.099695 18325 raft_consensus.cc:686] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 1 LEADER]: Becoming Leader. State: Replica: 9dd5b291ea0e44f8ab89bcf4a8433fd6, State: Running, Role: LEADER
I20260620 23:12:50.099805 18325 consensus_queue.cc:237] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } }
I20260620 23:12:50.100134 18361 tablet_replica.cc:439] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: TxnStatusTablet state changed. Reason: New leader 9dd5b291ea0e44f8ab89bcf4a8433fd6. Latest consensus state: current_term: 1 leader_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } }
I20260620 23:12:50.100302 18361 tablet_replica.cc:442] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:12:50.100512 17811 catalog_manager.cc:5495] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd5b291ea0e44f8ab89bcf4a8433fd6 (127.16.79.2). New cstate: current_term: 1 leader_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } health_report { overall_health: UNKNOWN } } }
I20260620 23:12:50.100574 18363 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:12:50.100766 18363 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:12:50.121110 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:50.118173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:46801 (local address 127.0.0.1:53324)
0620 23:12:50.118289 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:50.118342 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:50.118400 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:50.118597 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:50.118598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:50.118599 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:12:50.118670 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:50.118673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.119464 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.119464 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:50.120086 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:50.120087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.120480 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.120481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:50.120539 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:50.120557 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:12:50.120901 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:12:50.120912 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:12:50.120961 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 23:12:50.121033 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:12:50.121132 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:50.118268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.16.79.2:46801)
0620 23:12:50.118369 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:50.118372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:50.118382 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:12:50.118486 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:50.118487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:50.118516 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:50.118591 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:50.118752 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.119366 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.120172 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.120395 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.120478 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:50.120675 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:50.120828 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:50.120919 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:50.120971 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 23:12:50.121006 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:12:50.122514 18248 raft_consensus.cc:1261] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0 [term 1 FOLLOWER]: Refusing update from remote peer 9dd5b291ea0e44f8ab89bcf4a8433fd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:50.122623 17969 raft_consensus.cc:1261] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 FOLLOWER]: Refusing update from remote peer 9dd5b291ea0e44f8ab89bcf4a8433fd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:50.122956 18361 consensus_queue.cc:1035] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:50.123113 18325 consensus_queue.cc:1035] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:50.123826 18341 tablet_replica.cc:439] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1: TxnStatusTablet state changed. Reason: New leader 9dd5b291ea0e44f8ab89bcf4a8433fd6. Latest consensus state: current_term: 1 leader_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } }
I20260620 23:12:50.123930 18341 tablet_replica.cc:442] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:50.124183 18324 tablet_replica.cc:439] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0: TxnStatusTablet state changed. Reason: New leader 9dd5b291ea0e44f8ab89bcf4a8433fd6. Latest consensus state: current_term: 1 leader_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } }
I20260620 23:12:50.124281 18324 tablet_replica.cc:442] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:50.124796 18361 tablet_replica.cc:439] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } }
I20260620 23:12:50.124913 18361 tablet_replica.cc:442] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:12:50.125481 18324 tablet_replica.cc:439] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } }
I20260620 23:12:50.125564 18324 tablet_replica.cc:442] T 96fa6bce648745a9b8c764a3fc11dc52 P d032dc9235dc427e8301469bc635adb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:50.126348 18325 tablet_replica.cc:439] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } }
I20260620 23:12:50.126443 18325 tablet_replica.cc:442] T 96fa6bce648745a9b8c764a3fc11dc52 P 9dd5b291ea0e44f8ab89bcf4a8433fd6: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:12:50.127059 18334 mvcc.cc:205] Tried to move back new op lower bound from 7299060408819343360 to 7299060408729554944. Current Snapshot: MvccSnapshot[applied={T|T < 7299060408819343360}]
I20260620 23:12:50.127147 18341 tablet_replica.cc:439] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5a0e58326e4de8b74586f2ff8514c1" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 42939 } } peers { permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 } } peers { permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 } } }
I20260620 23:12:50.127255 18341 tablet_replica.cc:442] T 96fa6bce648745a9b8c764a3fc11dc52 P ef5a0e58326e4de8b74586f2ff8514c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:50.132422 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:50.129542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:39759 (local address 127.16.79.2:46801)
0620 23:12:50.129596 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:50.129598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:50.129772 (+ 174us) server_negotiation.cc:407] Connection header received
0620 23:12:50.129808 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:50.129809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:50.129820 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:50.129878 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:50.130059 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.130550 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.131693 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.131887 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.131948 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:50.132182 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:50.132213 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:50.132267 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:50.132345 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 23:12:50.132362 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:12:50.132468 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:50.129424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:46801 (local address 127.16.79.1:39759)
0620 23:12:50.129630 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:50.129642 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:50.129716 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:50.129895 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:50.129896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:50.129899 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:12:50.129950 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:50.129951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.130631 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.130631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:50.131586 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:50.131588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.131966 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.131967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:50.132026 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:50.132041 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:12:50.132284 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:12:50.132286 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:12:50.132340 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:12:50.132389 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":128,"threads_started":1}
I20260620 23:12:50.134627 18248 raft_consensus.cc:1261] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0 [term 1 FOLLOWER]: Refusing update from remote peer ef5a0e58326e4de8b74586f2ff8514c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:50.134646 18109 raft_consensus.cc:1261] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 1 FOLLOWER]: Refusing update from remote peer ef5a0e58326e4de8b74586f2ff8514c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:50.134850 18342 consensus_queue.cc:1035] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d032dc9235dc427e8301469bc635adb0" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:50.134940 18341 consensus_queue.cc:1035] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd5b291ea0e44f8ab89bcf4a8433fd6" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:50.136375 18335 mvcc.cc:205] Tried to move back new op lower bound from 7299060408869994496 to 7299060407906037760. Current Snapshot: MvccSnapshot[applied={T|T < 7299060408869994496}]
I20260620 23:12:50.143584 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:50.140835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:39251 (local address 127.16.79.1:42939)
0620 23:12:50.140873 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:50.140875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:50.140996 (+ 121us) server_negotiation.cc:407] Connection header received
0620 23:12:50.141100 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:50.141101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:50.141119 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:50.141160 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:50.141279 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.141715 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.142850 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.143148 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.143186 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:50.143402 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:50.143430 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:50.143471 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:50.143523 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 23:12:50.143539 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:12:50.143654 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:50.140633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:42939 (local address 127.16.79.2:39251)
0620 23:12:50.140939 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:50.140950 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:50.141020 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:50.141164 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:50.141165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:50.141166 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:12:50.141223 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:50.141224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.141781 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.141782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:50.142749 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:50.142751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.143249 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.143250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:50.143310 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:50.143325 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:12:50.143472 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:12:50.143473 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:12:50.143522 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 23:12:50.143562 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
I20260620 23:12:50.152472 17811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:50.152586 17811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:50.153235 17811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa454770d5784e809fbd5bda43b5006b: Sending DeleteTablet for 3 replicas of tablet c10ce164f70140adbbdd86475bb1e18a
I20260620 23:12:50.153482 18089 tablet_service.cc:1514] Processing DeleteTablet for tablet c10ce164f70140adbbdd86475bb1e18a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:50 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:53320
I20260620 23:12:50.153486 18229 tablet_service.cc:1514] Processing DeleteTablet for tablet c10ce164f70140adbbdd86475bb1e18a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:50 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:40678
I20260620 23:12:50.153486 17949 tablet_service.cc:1514] Processing DeleteTablet for tablet c10ce164f70140adbbdd86475bb1e18a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:50 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:45838
I20260620 23:12:50.153580 18345 tablet_replica.cc:330] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0: stopping tablet replica
I20260620 23:12:50.153582 18347 tablet_replica.cc:330] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6: stopping tablet replica
I20260620 23:12:50.153582 18346 tablet_replica.cc:330] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1: stopping tablet replica
I20260620 23:12:50.153627 18345 raft_consensus.cc:2229] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:50.153627 18347 raft_consensus.cc:2229] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:50.153636 18346 raft_consensus.cc:2229] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:12:50.153662 18345 raft_consensus.cc:2258] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:50.153662 18347 raft_consensus.cc:2258] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:50.153685 18346 raft_consensus.cc:2258] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:50.153954 18347 ts_tablet_manager.cc:1905] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:50.153968 18346 ts_tablet_manager.cc:1905] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:50.153975 18345 ts_tablet_manager.cc:1905] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:50.155889 18346 ts_tablet_manager.cc:1918] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:12:50.155902 18347 ts_tablet_manager.cc:1918] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:12:50.155946 18346 log.cc:1196] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-0/wal/wals/c10ce164f70140adbbdd86475bb1e18a
I20260620 23:12:50.155969 18347 log.cc:1196] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/c10ce164f70140adbbdd86475bb1e18a
I20260620 23:12:50.156169 18346 ts_tablet_manager.cc:1939] T c10ce164f70140adbbdd86475bb1e18a P ef5a0e58326e4de8b74586f2ff8514c1: Deleting consensus metadata
I20260620 23:12:50.156206 18347 ts_tablet_manager.cc:1939] T c10ce164f70140adbbdd86475bb1e18a P 9dd5b291ea0e44f8ab89bcf4a8433fd6: Deleting consensus metadata
I20260620 23:12:50.156455 18345 ts_tablet_manager.cc:1918] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:12:50.156476 17796 catalog_manager.cc:4841] TS ef5a0e58326e4de8b74586f2ff8514c1 (127.16.79.1:42939): tablet c10ce164f70140adbbdd86475bb1e18a (table test-table [id=021e3bb95d94426fb59e47905efa4678]) successfully deleted
I20260620 23:12:50.156512 18345 log.cc:1196] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/c10ce164f70140adbbdd86475bb1e18a
I20260620 23:12:50.156608 17798 catalog_manager.cc:4841] TS 9dd5b291ea0e44f8ab89bcf4a8433fd6 (127.16.79.2:46801): tablet c10ce164f70140adbbdd86475bb1e18a (table test-table [id=021e3bb95d94426fb59e47905efa4678]) successfully deleted
I20260620 23:12:50.156755 18345 ts_tablet_manager.cc:1939] T c10ce164f70140adbbdd86475bb1e18a P d032dc9235dc427e8301469bc635adb0: Deleting consensus metadata
I20260620 23:12:50.157089 17796 catalog_manager.cc:4841] TS d032dc9235dc427e8301469bc635adb0 (127.16.79.3:34277): tablet c10ce164f70140adbbdd86475bb1e18a (table test-table [id=021e3bb95d94426fb59e47905efa4678]) successfully deleted
Jun 20 23:12:50 dist-test-slave-pswb krb5kdc[17767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997169, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.79.1@KRBTEST.COM
I20260620 23:12:50.165112 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:50.155889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.16.79.1:42939)
0620 23:12:50.155926 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:50.155928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:50.157099 (+ 1171us) server_negotiation.cc:407] Connection header received
0620 23:12:50.157194 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:50.157194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:50.157204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:50.157231 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:50.157802 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.158246 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.158964 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.159150 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.159225 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:50.162212 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:50.162219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:50.162219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:50.162228 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:50.163810 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:50.164174 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:50.164177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:50.164178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:50.164245 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:50.164517 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:50.164518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:50.164519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:50.164655 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:50.164830 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:50.164934 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 23:12:50.165007 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 23:12:50.165315 18009 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45866
I20260620 23:12:50.173743 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:50.166178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.16.79.62:33041)
0620 23:12:50.166241 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:50.166243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:50.166395 (+ 152us) server_negotiation.cc:407] Connection header received
0620 23:12:50.166465 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:50.166465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:50.166474 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:50.166517 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:50.167395 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.168153 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.169150 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:50.169352 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:50.169410 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:50.170750 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:50.170752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:50.170752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:50.170762 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:50.172561 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:50.173010 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:50.173011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:50.173012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:50.173079 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:50.173293 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:50.173293 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:50.173293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:50.173402 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:50.173478 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:50.173615 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:12:50.173685 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 23:12:50.174120 17811 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56262
I20260620 23:12:50.174669 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 17884
I20260620 23:12:50.181535 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 18024
I20260620 23:12:50.186772 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 18164
I20260620 23:12:50.191445 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 17780
2026-06-20T23:12:50Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2555 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:12:50 dist-test-slave-pswb krb5kdc[18384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:12:50 dist-test-slave-pswb krb5kdc[18384](info): set up 2 sockets
Jun 20 23:12:50 dist-test-slave-pswb krb5kdc[18384](info): commencing operation
krb5kdc: starting...
W20260620 23:12:51.240976 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:12:51 dist-test-slave-pswb krb5kdc[18384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:12:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:12:51.351467 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:46409
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:46409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:51.431936 18397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:51.432158 18397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:51.432190 18397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:51.433310 18397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:12:51.433372 18397 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:12:51.433394 18397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:51.433418 18397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:51.433446 18397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:12:51.433470 18397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:12:51.434882 18397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:46409
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:46409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:51.435148 18397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:51.435364 18397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:51.440214 18403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:51.440214 18405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:51.440246 18397 server_base.cc:1034] running on GCE node
W20260620 23:12:51.440258 18402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:51.441447 18397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:51.441763 18397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:51.442942 18397 hybrid_clock.cc:648] HybridClock initialized: now 1781997171442928 us; error 26 us; skew 500 ppm
Jun 20 23:12:51 dist-test-slave-pswb krb5kdc[18384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997171, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:51.445591 18397 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:12:51.446033 18397 webserver.cc:460] Webserver started at http://127.16.79.62:37029/ using document root <none> and password file <none>
I20260620 23:12:51.446278 18397 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:51.446328 18397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:51.446451 18397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:51.447548 18397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "9b653a0aa1cd41619320f280704b6594"
format_stamp: "Formatted at 2026-06-20 23:12:51 on dist-test-slave-pswb"
server_key: "a6dc0ed3a97d5b6b512009905d9cf154"
server_key_iv: "6f45d7a97a9d264c767d6cabbbc499be"
server_key_version: "encryptionkey@0"
I20260620 23:12:51.447939 18397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "9b653a0aa1cd41619320f280704b6594"
format_stamp: "Formatted at 2026-06-20 23:12:51 on dist-test-slave-pswb"
server_key: "a6dc0ed3a97d5b6b512009905d9cf154"
server_key_iv: "6f45d7a97a9d264c767d6cabbbc499be"
server_key_version: "encryptionkey@0"
I20260620 23:12:51.449419 18397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:12:51.450193 18412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:51.450373 18397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:12:51.450480 18397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "9b653a0aa1cd41619320f280704b6594"
format_stamp: "Formatted at 2026-06-20 23:12:51 on dist-test-slave-pswb"
server_key: "a6dc0ed3a97d5b6b512009905d9cf154"
server_key_iv: "6f45d7a97a9d264c767d6cabbbc499be"
server_key_version: "encryptionkey@0"
I20260620 23:12:51.450567 18397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:51.467283 18397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:51.469813 18397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:51.469985 18397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:51.474982 18397 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:46409
I20260620 23:12:51.475031 18474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:46409 every 8 connection(s)
I20260620 23:12:51.475365 18397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:12:51.475919 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 18397
I20260620 23:12:51.475971 18475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:51.476029 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:12:51.476203 16700 external_mini_cluster.cc:1428] Setting key 8cf624f9835771417b0a23ba77b6db7e
I20260620 23:12:51.479205 18475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594: Bootstrap starting.
I20260620 23:12:51.479859 18475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:51.480142 18475 log.cc:824] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:51.480923 18475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594: No bootstrap required, opened a new log
Jun 20 23:12:51 dist-test-slave-pswb krb5kdc[18384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:51.483632 18475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b653a0aa1cd41619320f280704b6594" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46409 } }
I20260620 23:12:51.483794 18475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:51.483850 18475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b653a0aa1cd41619320f280704b6594, State: Initialized, Role: FOLLOWER
I20260620 23:12:51.484030 18475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b653a0aa1cd41619320f280704b6594" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46409 } }
I20260620 23:12:51.484129 18475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:12:51.484165 18475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:12:51.484229 18475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:51.484930 18475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b653a0aa1cd41619320f280704b6594" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46409 } }
I20260620 23:12:51.485067 18475 leader_election.cc:302] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b653a0aa1cd41619320f280704b6594; no voters:
I20260620 23:12:51.485265 18475 leader_election.cc:288] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:12:51.485293 18480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:51.485421 18480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [term 1 LEADER]: Becoming Leader. State: Replica: 9b653a0aa1cd41619320f280704b6594, State: Running, Role: LEADER
I20260620 23:12:51.485525 18475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:12:51.485562 18480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b653a0aa1cd41619320f280704b6594" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46409 } }
I20260620 23:12:51.485968 18480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b653a0aa1cd41619320f280704b6594. Latest consensus state: current_term: 1 leader_uuid: "9b653a0aa1cd41619320f280704b6594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b653a0aa1cd41619320f280704b6594" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46409 } } }
I20260620 23:12:51.486205 18480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:51.486131 18482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b653a0aa1cd41619320f280704b6594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b653a0aa1cd41619320f280704b6594" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46409 } } }
I20260620 23:12:51.486647 18482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594 [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:51.486819 18489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:12:51.487309 18489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:12:51.487556 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:51.476769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.16.79.62:46409)
0620 23:12:51.476970 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:51.476979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:51.477011 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:12:51.477499 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:51.477508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:51.477569 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:51.477729 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:51.478324 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.479655 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.480533 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.481025 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.481135 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:51.483799 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:51.483803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:51.483803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:51.483812 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:51.485835 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:51.486255 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:51.486258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:51.486259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:51.486368 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:51.486586 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:51.486587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:51.486588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:51.486890 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:51.487026 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:51.487188 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 23:12:51.487332 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
I20260620 23:12:51.489262 18489 catalog_manager.cc:1347] Generated new cluster ID: ad502a145135407bae15135ad62832e4
I20260620 23:12:51.489318 18489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:12:51.519682 18489 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:12:51.520210 18489 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:12:51.522907 18489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594: Generated new TSK 0
I20260620 23:12:51.523077 18489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:12:51.555689 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46409
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:51.636224 18501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:51.636411 18501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:51.636435 18501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:51.637602 18501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:51.637645 18501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:51.637666 18501 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:12:51.637678 18501 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:12:51.637733 18501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:12:51.639113 18501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.79.62:46409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:51.639351 18501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:51.639580 18501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:51.644418 18507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:51.644470 18509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:51.644419 18506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:51.644884 18501 server_base.cc:1034] running on GCE node
I20260620 23:12:51.645099 18501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:51.645334 18501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:51.646473 18501 hybrid_clock.cc:648] HybridClock initialized: now 1781997171646452 us; error 36 us; skew 500 ppm
Jun 20 23:12:51 dist-test-slave-pswb krb5kdc[18384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997171, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:51.648870 18501 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:12:51.649293 18501 webserver.cc:460] Webserver started at http://127.16.79.1:34369/ using document root <none> and password file <none>
I20260620 23:12:51.649478 18501 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:51.649518 18501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:51.649601 18501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:51.650517 18501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "a1316ced4e934746b29fba8e79522b9f"
format_stamp: "Formatted at 2026-06-20 23:12:51 on dist-test-slave-pswb"
server_key: "e1255041aeaf1ca92658a3fe6e907593"
server_key_iv: "912f273539c47f354250272427274c98"
server_key_version: "encryptionkey@0"
I20260620 23:12:51.650794 18501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "a1316ced4e934746b29fba8e79522b9f"
format_stamp: "Formatted at 2026-06-20 23:12:51 on dist-test-slave-pswb"
server_key: "e1255041aeaf1ca92658a3fe6e907593"
server_key_iv: "912f273539c47f354250272427274c98"
server_key_version: "encryptionkey@0"
I20260620 23:12:51.652092 18501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:12:51.652969 18516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:51.653136 18501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:51.653230 18501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "a1316ced4e934746b29fba8e79522b9f"
format_stamp: "Formatted at 2026-06-20 23:12:51 on dist-test-slave-pswb"
server_key: "e1255041aeaf1ca92658a3fe6e907593"
server_key_iv: "912f273539c47f354250272427274c98"
server_key_version: "encryptionkey@0"
I20260620 23:12:51.653309 18501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:51.661211 18501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:51.662951 18501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:51.663101 18501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:51.663605 18501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:51.663662 18501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:51.663733 18501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:51.663782 18501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:12:51 dist-test-slave-pswb krb5kdc[18384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997171, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:51.670979 18501 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:34503
I20260620 23:12:51.671046 18631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:34503 every 8 connection(s)
I20260620 23:12:51.671353 18501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:12:51.673563 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:51.663937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:35007 (local address 127.16.79.62:46409)
0620 23:12:51.663983 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:51.663986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:51.664244 (+ 258us) server_negotiation.cc:407] Connection header received
0620 23:12:51.664933 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:51.664934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:51.664944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:51.664986 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:51.665817 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.666334 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.667328 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.667568 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.667646 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:51.670947 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:51.670949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:51.670949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:51.670956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:51.672467 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:51.672873 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:51.672874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:51.672874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:51.672920 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:51.673078 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:51.673079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:51.673079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:51.673166 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:51.673211 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:51.673439 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 23:12:51.673502 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:12:51.673767 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:51.663933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46409 (local address 127.16.79.1:35007)
0620 23:12:51.664106 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:51.664144 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:51.664827 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:51.665056 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:51.665059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:51.665417 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:51.665714 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:51.665720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.666445 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.666446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:51.667219 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:51.667221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.667666 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.667667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:51.667834 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:51.668332 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:51.668342 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:51.670860 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:51.672537 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:51.672541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:51.672542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:51.672815 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:51.672964 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:51.672964 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:51.672965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:51.673042 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:51.673217 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:51.673221 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:51.673316 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 23:12:51.673422 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:12:51.673555 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260620 23:12:51.674242 18632 heartbeater.cc:347] Connected to a master server at 127.16.79.62:46409
I20260620 23:12:51.674317 18632 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:51.674528 18632 heartbeater.cc:511] Master 127.16.79.62:46409 requested a full tablet report, sending...
I20260620 23:12:51.675005 18428 ts_manager.cc:194] Registered new tserver with Master: a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503)
I20260620 23:12:51.675745 18428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:35007
I20260620 23:12:51.680819 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:51.675370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:39065 (local address 127.16.79.62:46409)
0620 23:12:51.675405 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:51.675407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:51.675529 (+ 122us) server_negotiation.cc:407] Connection header received
0620 23:12:51.675599 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:51.675600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:51.675610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:51.675664 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:51.676012 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.676497 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.677188 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.677358 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.677412 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:51.678323 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:51.678326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:51.678326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:51.678353 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:51.679675 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:51.679969 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:51.679970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:51.679970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:51.680018 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:51.680171 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:51.680172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:51.680172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:51.680257 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:51.680304 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:51.680608 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 23:12:51.680744 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:12:51.681020 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:51.675306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46409 (local address 127.16.79.1:39065)
0620 23:12:51.675465 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:51.675474 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:51.675543 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:51.675668 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:51.675669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:51.675839 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:51.675883 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:51.675885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.676586 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.676586 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:51.677115 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:51.677117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.677422 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.677423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:51.677475 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:51.677894 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:51.677899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:51.678152 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:51.679749 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:51.679750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:51.679751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:51.679903 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:51.680073 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:51.680073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:51.680074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:51.680121 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:51.680375 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:51.680377 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:51.680450 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 23:12:51.680553 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 23:12:51.680769 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260620 23:12:51.681326 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 18501
I20260620 23:12:51.681432 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:12:51.681597 16700 external_mini_cluster.cc:1428] Setting key cb0f7a6b848536830c7289d444ba5fb9
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:12:51.706995 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46409
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:51.786259 18641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:51.786478 18641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:51.786504 18641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:51.787638 18641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:51.787678 18641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:51.787698 18641 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:12:51.787710 18641 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:12:51.787727 18641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:12:51.789055 18641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.79.62:46409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:51.789240 18641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:51.789398 18641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:51.794198 18649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:51.794209 18647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:51.794421 18646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:51.794322 18641 server_base.cc:1034] running on GCE node
I20260620 23:12:51.794654 18641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:51.794836 18641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:51.795979 18641 hybrid_clock.cc:648] HybridClock initialized: now 1781997171795969 us; error 26 us; skew 500 ppm
Jun 20 23:12:51 dist-test-slave-pswb krb5kdc[18384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997171, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:51.798785 18641 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:12:51.799232 18641 webserver.cc:460] Webserver started at http://127.16.79.2:38065/ using document root <none> and password file <none>
I20260620 23:12:51.799489 18641 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:51.799558 18641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:51.799705 18641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:51.800685 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "0eb8ca9ef5de4435bde4bec9d837acd2"
format_stamp: "Formatted at 2026-06-20 23:12:51 on dist-test-slave-pswb"
server_key: "66d9b902d456b27e173e32621669db9c"
server_key_iv: "4c8a5aac31f49c1aff72a560b1153356"
server_key_version: "encryptionkey@0"
I20260620 23:12:51.801075 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "0eb8ca9ef5de4435bde4bec9d837acd2"
format_stamp: "Formatted at 2026-06-20 23:12:51 on dist-test-slave-pswb"
server_key: "66d9b902d456b27e173e32621669db9c"
server_key_iv: "4c8a5aac31f49c1aff72a560b1153356"
server_key_version: "encryptionkey@0"
I20260620 23:12:51.802356 18641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:51.803165 18656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:51.803359 18641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:51.803459 18641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "0eb8ca9ef5de4435bde4bec9d837acd2"
format_stamp: "Formatted at 2026-06-20 23:12:51 on dist-test-slave-pswb"
server_key: "66d9b902d456b27e173e32621669db9c"
server_key_iv: "4c8a5aac31f49c1aff72a560b1153356"
server_key_version: "encryptionkey@0"
I20260620 23:12:51.803539 18641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:51.826161 18641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:51.828384 18641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:51.828543 18641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:51.829175 18641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:51.829231 18641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:51.829264 18641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:51.829281 18641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:12:51 dist-test-slave-pswb krb5kdc[18384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997171, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:51.837522 18641 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:34951
I20260620 23:12:51.837585 18771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:34951 every 8 connection(s)
I20260620 23:12:51.837958 18641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:12:51.838567 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:51.829681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:39687 (local address 127.16.79.62:46409)
0620 23:12:51.829752 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:51.829755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:51.829943 (+ 188us) server_negotiation.cc:407] Connection header received
0620 23:12:51.830390 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:51.830391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:51.830404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:51.830485 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:51.831116 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.831908 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.833194 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.833411 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.833494 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:51.835668 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:51.835669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:51.835669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:51.835675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:51.837134 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:51.837627 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:51.837628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:51.837629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:51.837697 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:51.837941 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:51.837942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:51.837943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:51.838080 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:51.838161 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:51.838427 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:12:51.838505 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:12:51.838714 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:51.829643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46409 (local address 127.16.79.2:39687)
0620 23:12:51.829814 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:51.829852 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:51.830304 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:51.830548 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:51.830551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:51.830842 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:51.831013 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:51.831017 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.831999 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.832000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:51.833087 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:51.833089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.833504 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.833504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:51.833632 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:51.834078 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:51.834086 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:51.835579 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:51.837213 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:51.837218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:51.837222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:51.837537 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:51.837779 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:51.837780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:51.837780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:51.837870 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:51.838147 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:51.838153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:51.838294 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 23:12:51.838403 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:12:51.838535 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260620 23:12:51.839140 18772 heartbeater.cc:347] Connected to a master server at 127.16.79.62:46409
I20260620 23:12:51.839217 18772 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:51.839380 18772 heartbeater.cc:511] Master 127.16.79.62:46409 requested a full tablet report, sending...
I20260620 23:12:51.839730 18428 ts_manager.cc:194] Registered new tserver with Master: 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2:34951)
I20260620 23:12:51.840221 18428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:39687
I20260620 23:12:51.841358 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 18641
I20260620 23:12:51.841449 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:12:51.841629 16700 external_mini_cluster.cc:1428] Setting key 4cf39328fe7c98543d1418483c43f1b6
I20260620 23:12:51.846300 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:51.840145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:38881 (local address 127.16.79.62:46409)
0620 23:12:51.840185 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:51.840187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:51.840250 (+ 63us) server_negotiation.cc:407] Connection header received
0620 23:12:51.840322 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:51.840323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:51.840331 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:51.840378 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:51.840859 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.841647 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.842426 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.842728 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.842815 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:51.843815 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:51.843816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:51.843817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:51.843822 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:51.845163 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:51.845512 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:51.845513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:51.845513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:51.845568 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:51.845774 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:51.845775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:51.845775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:51.845919 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:51.845996 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:51.846143 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 23:12:51.846214 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:12:51.846315 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:51.840062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46409 (local address 127.16.79.2:38881)
0620 23:12:51.840193 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:51.840203 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:51.840262 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:51.840395 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:51.840395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:51.840601 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:51.840670 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:51.840672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.841763 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.841764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:51.842352 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:51.842354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:51.842810 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:51.842811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:51.842904 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:51.843332 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:51.843336 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:51.843723 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:51.845266 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:51.845267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:51.845268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:51.845437 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:51.845642 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:51.845643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:51.845643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:51.845700 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:51.845988 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:51.845989 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:51.846032 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 23:12:51.846120 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:12:51.846218 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:12:51.865583 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46409
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:51.947103 18781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:51.947290 18781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:51.947314 18781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:51.948432 18781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:51.948472 18781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:51.948493 18781 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:12:51.948504 18781 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:12:51.948546 18781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:12:51.950047 18781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.79.62:46409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:51.950311 18781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:51.950527 18781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:51.955435 18787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:51.955435 18786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:51.955642 18781 server_base.cc:1034] running on GCE node
W20260620 23:12:51.955670 18789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:51.955909 18781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:51.956139 18781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:51.957314 18781 hybrid_clock.cc:648] HybridClock initialized: now 1781997171957297 us; error 30 us; skew 500 ppm
Jun 20 23:12:51 dist-test-slave-pswb krb5kdc[18384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997171, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:51.959888 18781 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:12:51.960340 18781 webserver.cc:460] Webserver started at http://127.16.79.3:39989/ using document root <none> and password file <none>
I20260620 23:12:51.960577 18781 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:51.960623 18781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:51.960763 18781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:51.962122 18781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "a80a26c5c8c745249b0786fb3f708b69"
format_stamp: "Formatted at 2026-06-20 23:12:51 on dist-test-slave-pswb"
server_key: "b9273d8f981738a13d89189beac85f5b"
server_key_iv: "d887d7725c453f7f16efaeba6144648a"
server_key_version: "encryptionkey@0"
I20260620 23:12:51.962529 18781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "a80a26c5c8c745249b0786fb3f708b69"
format_stamp: "Formatted at 2026-06-20 23:12:51 on dist-test-slave-pswb"
server_key: "b9273d8f981738a13d89189beac85f5b"
server_key_iv: "d887d7725c453f7f16efaeba6144648a"
server_key_version: "encryptionkey@0"
I20260620 23:12:51.967149 18781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 23:12:51.968808 18796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:51.968961 18781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:51.969065 18781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "a80a26c5c8c745249b0786fb3f708b69"
format_stamp: "Formatted at 2026-06-20 23:12:51 on dist-test-slave-pswb"
server_key: "b9273d8f981738a13d89189beac85f5b"
server_key_iv: "d887d7725c453f7f16efaeba6144648a"
server_key_version: "encryptionkey@0"
I20260620 23:12:51.969158 18781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:52.001876 18781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:52.003798 18781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:52.003988 18781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:52.004549 18781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:52.004596 18781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:52.004627 18781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:52.004724 18781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:12:52 dist-test-slave-pswb krb5kdc[18384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997171, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:52.011744 18781 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:40837
I20260620 23:12:52.011802 18911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:40837 every 8 connection(s)
I20260620 23:12:52.012151 18781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:12:52.014671 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.004905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:39787 (local address 127.16.79.62:46409)
0620 23:12:52.004948 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.004950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.005244 (+ 294us) server_negotiation.cc:407] Connection header received
0620 23:12:52.005819 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.005820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.005834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.005914 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:52.006631 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.007440 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.008437 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.008808 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.008882 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.011384 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:52.011386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:52.011386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:52.011393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:52.013198 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:52.013560 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:52.013561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:52.013561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:52.013617 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:52.013931 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:52.013934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:52.013934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:52.014133 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:52.014241 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.014488 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.014592 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:12:52.014700 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.004912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46409 (local address 127.16.79.3:39787)
0620 23:12:52.005084 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.005133 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.005711 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.006016 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.006020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.006380 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:52.006554 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.006559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.007544 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.007545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.008334 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.008336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.008922 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.008923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.009036 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.009500 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:52.009509 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:52.011295 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:52.013267 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:52.013272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:52.013274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:52.013475 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:52.013687 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:52.013687 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:52.013687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:52.013771 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:52.014242 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:52.014247 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:52.014355 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.014446 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.014555 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260620 23:12:52.015298 18912 heartbeater.cc:347] Connected to a master server at 127.16.79.62:46409
I20260620 23:12:52.015391 18912 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:52.015556 18912 heartbeater.cc:511] Master 127.16.79.62:46409 requested a full tablet report, sending...
I20260620 23:12:52.015941 18429 ts_manager.cc:194] Registered new tserver with Master: a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3:40837)
I20260620 23:12:52.016441 18429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:39787
I20260620 23:12:52.020272 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 18781
I20260620 23:12:52.020361 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:12:52.020548 16700 external_mini_cluster.cc:1428] Setting key 930d17a5b23d128b17a332b1c0e27571
I20260620 23:12:52.021411 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:12:52.021862 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.016263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:53961 (local address 127.16.79.62:46409)
0620 23:12:52.016300 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.016301 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.016426 (+ 125us) server_negotiation.cc:407] Connection header received
0620 23:12:52.016496 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.016497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.016506 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.016553 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:52.017005 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.017456 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.018164 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.018341 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.018390 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.019185 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:52.019186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:52.019187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:52.019192 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:52.020667 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:52.021091 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:52.021092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:52.021093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:52.021154 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:52.021375 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:52.021376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:52.021377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:52.021530 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:52.021602 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.021738 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.021800 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:12:52.021921 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.016200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46409 (local address 127.16.79.3:53961)
0620 23:12:52.016361 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.016370 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.016439 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.016593 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.016594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.016843 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:52.016915 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.016917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.017569 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.017570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.018091 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.018093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.018419 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.018420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.018510 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.018842 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:52.018846 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:52.019110 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:52.020784 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:52.020785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:52.020785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:52.020995 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:52.021243 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:52.021243 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:52.021244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:52.021308 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:52.021599 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:52.021600 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:52.021645 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.021721 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.021815 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
Jun 20 23:12:52 dist-test-slave-pswb krb5kdc[18384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:12:52 dist-test-slave-pswb krb5kdc[18384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:52.045274 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.037550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.16.79.62:46409)
0620 23:12:52.037642 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.037644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.037653 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:12:52.037679 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.037685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.037694 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.037748 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:52.038329 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.038891 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.039770 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.039972 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.040033 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.042713 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:52.042715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:52.042716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:52.042725 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:52.044167 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:52.044552 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:52.044553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:52.044553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:52.044615 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:52.044869 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:52.044870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:52.044870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:52.044973 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:52.045051 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.045152 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.045214 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:12:52.046993 18429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:12:52.048497 18429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:12:52.055727 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.051491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.16.79.3:40837)
0620 23:12:52.051712 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.051717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.051727 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:12:52.051781 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.051783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.051799 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.051881 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:52.052082 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.053140 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.054459 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.055217 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.055296 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.055358 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.055598 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.055636 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
I20260620 23:12:52.055734 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.050991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:40837 (local address 127.0.0.1:44060)
0620 23:12:52.051197 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.051211 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.051286 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.051952 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.051953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.051956 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:52.051997 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.051998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.053198 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.053199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.054371 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.054374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.055351 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.055352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.055470 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.055488 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.055576 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.055631 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":49,"thread_start_us":128,"threads_started":1}
I20260620 23:12:52.055735 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.050786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34951 (local address 127.0.0.1:40976)
0620 23:12:52.050914 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.050953 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.051051 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.051681 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.051685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.051688 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:52.051777 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.051778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.053133 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.053134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.054543 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.054545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.055397 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.055398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.055470 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.055488 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.055567 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.055632 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":70,"thread_start_us":85,"threads_started":1}
I20260620 23:12:52.056453 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.051270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.16.79.2:34951)
0620 23:12:52.051434 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.051438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.051451 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:12:52.051493 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.051495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.051522 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.051625 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:52.051892 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.053041 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.054638 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.055338 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.055396 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.056301 (+ 905us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.056326 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.056364 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260620 23:12:52.056933 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.051070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:34503 (local address 127.0.0.1:57760)
0620 23:12:52.051215 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.051237 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.051286 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.052146 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.052147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.052149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:52.052211 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.052213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.053184 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.053185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.054541 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.054544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.056610 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.056611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.056741 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.056747 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.056801 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.056851 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":56,"thread_start_us":109,"threads_started":1}
I20260620 23:12:52.056717 18846 tablet_service.cc:1467] Processing CreateTablet for tablet f0f39b9095e34fa5ab0a0b2eee9e09dd (DEFAULT_TABLE table=test-table [id=84d103106b704fe2ab61abe19ad0dbbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:52.057195 18846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f39b9095e34fa5ab0a0b2eee9e09dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:52.057261 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.051792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.16.79.1:34503)
0620 23:12:52.051976 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.051980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.051991 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:12:52.052032 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.052034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.052050 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.052131 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:52.052270 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.053058 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.055451 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.056168 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.056606 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.056667 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.057116 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.057160 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260620 23:12:52.057433 18706 tablet_service.cc:1467] Processing CreateTablet for tablet f0f39b9095e34fa5ab0a0b2eee9e09dd (DEFAULT_TABLE table=test-table [id=84d103106b704fe2ab61abe19ad0dbbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:52.057770 18706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f39b9095e34fa5ab0a0b2eee9e09dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:52.060097 18935 tablet_bootstrap.cc:492] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69: Bootstrap starting.
I20260620 23:12:52.060585 18936 tablet_bootstrap.cc:492] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2: Bootstrap starting.
I20260620 23:12:52.061273 18936 tablet_bootstrap.cc:654] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:52.061410 18935 tablet_bootstrap.cc:654] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:52.061367 18548 tablet_service.cc:1467] Processing CreateTablet for tablet f0f39b9095e34fa5ab0a0b2eee9e09dd (DEFAULT_TABLE table=test-table [id=84d103106b704fe2ab61abe19ad0dbbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:52.061641 18936 log.cc:824] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:52.061708 18548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f39b9095e34fa5ab0a0b2eee9e09dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:52.061725 18935 log.cc:824] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:52.062352 18936 tablet_bootstrap.cc:492] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2: No bootstrap required, opened a new log
I20260620 23:12:52.062439 18936 ts_tablet_manager.cc:1397] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:12:52.064057 18936 raft_consensus.cc:348] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.064229 18936 raft_consensus.cc:374] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:52.064276 18936 raft_consensus.cc:729] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb8ca9ef5de4435bde4bec9d837acd2, State: Initialized, Role: FOLLOWER
I20260620 23:12:52.064407 18936 consensus_queue.cc:260] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.064648 18939 tablet_bootstrap.cc:492] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f: Bootstrap starting.
I20260620 23:12:52.064728 18936 ts_tablet_manager.cc:1428] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:12:52.064793 18772 heartbeater.cc:503] Master 127.16.79.62:46409 was elected leader, sending a full tablet report...
I20260620 23:12:52.065148 18935 tablet_bootstrap.cc:492] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69: No bootstrap required, opened a new log
I20260620 23:12:52.065220 18935 ts_tablet_manager.cc:1397] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 23:12:52.065397 18939 tablet_bootstrap.cc:654] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:52.065737 18939 log.cc:824] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:52.066504 18939 tablet_bootstrap.cc:492] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f: No bootstrap required, opened a new log
I20260620 23:12:52.066574 18939 ts_tablet_manager.cc:1397] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:12:52.066943 18935 raft_consensus.cc:348] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.067061 18935 raft_consensus.cc:374] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:52.067082 18935 raft_consensus.cc:729] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80a26c5c8c745249b0786fb3f708b69, State: Initialized, Role: FOLLOWER
I20260620 23:12:52.067185 18935 consensus_queue.cc:260] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.067427 18935 ts_tablet_manager.cc:1428] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:12:52.067466 18912 heartbeater.cc:503] Master 127.16.79.62:46409 was elected leader, sending a full tablet report...
I20260620 23:12:52.068598 18939 raft_consensus.cc:348] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.068794 18939 raft_consensus.cc:374] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:52.068845 18939 raft_consensus.cc:729] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1316ced4e934746b29fba8e79522b9f, State: Initialized, Role: FOLLOWER
I20260620 23:12:52.068957 18939 consensus_queue.cc:260] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.069247 18939 ts_tablet_manager.cc:1428] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:12:52.069280 18632 heartbeater.cc:503] Master 127.16.79.62:46409 was elected leader, sending a full tablet report...
W20260620 23:12:52.088476 18773 tablet.cc:2369] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:12:52.172048 18633 tablet.cc:2369] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:12:52.236343 18940 raft_consensus.cc:482] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:12:52.236580 18940 raft_consensus.cc:504] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.237120 18940 leader_election.cc:288] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503), a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3:40837)
I20260620 23:12:52.240288 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.237358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:40837 (local address 127.16.79.2:46309)
0620 23:12:52.237510 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.237522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.237574 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.237790 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.237791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.237793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:52.237868 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.237870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.238536 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.238537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.239347 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.239350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.240015 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.240016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.240099 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.240102 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.240169 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.240213 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 23:12:52.240288 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.237588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:46309 (local address 127.16.79.3:40837)
0620 23:12:52.237658 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.237660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.237668 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:12:52.237701 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.237701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.237710 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.237784 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:52.237941 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.238426 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.239443 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.239930 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.240005 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.240015 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.240174 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.240208 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:12:52.240788 18866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f39b9095e34fa5ab0a0b2eee9e09dd" candidate_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80a26c5c8c745249b0786fb3f708b69" is_pre_election: true
I20260620 23:12:52.240955 18866 raft_consensus.cc:2454] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0eb8ca9ef5de4435bde4bec9d837acd2 in term 0.
I20260620 23:12:52.241096 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.237358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:34503 (local address 127.16.79.2:49719)
0620 23:12:52.237655 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.237668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.237801 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.238133 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.238134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.238137 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:52.238239 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.238240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.239083 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.239084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.240038 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.240041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.240755 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.240757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.240843 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.240847 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.240928 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.240978 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":20,"thread_start_us":112,"threads_started":1}
I20260620 23:12:52.241083 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.237582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:49719 (local address 127.16.79.1:34503)
0620 23:12:52.237643 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.237646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.237778 (+ 132us) server_negotiation.cc:407] Connection header received
0620 23:12:52.237904 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.237904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.237917 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.238108 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:52.238314 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.238980 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.240129 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.240613 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.240741 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.240756 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.240970 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.241012 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 23:12:52.241182 18660 leader_election.cc:302] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eb8ca9ef5de4435bde4bec9d837acd2, a80a26c5c8c745249b0786fb3f708b69; no voters:
I20260620 23:12:52.241340 18940 raft_consensus.cc:2781] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:12:52.241395 18940 raft_consensus.cc:482] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:12:52.241420 18940 raft_consensus.cc:3037] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:52.241483 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f39b9095e34fa5ab0a0b2eee9e09dd" candidate_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1316ced4e934746b29fba8e79522b9f" is_pre_election: true
I20260620 23:12:52.241645 18586 raft_consensus.cc:2454] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0eb8ca9ef5de4435bde4bec9d837acd2 in term 0.
I20260620 23:12:52.242537 18940 raft_consensus.cc:504] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.242692 18940 leader_election.cc:288] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [CANDIDATE]: Term 1 election: Requested vote from peers a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503), a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3:40837)
I20260620 23:12:52.242889 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f39b9095e34fa5ab0a0b2eee9e09dd" candidate_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1316ced4e934746b29fba8e79522b9f"
I20260620 23:12:52.243016 18586 raft_consensus.cc:3037] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:52.243018 18866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f39b9095e34fa5ab0a0b2eee9e09dd" candidate_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80a26c5c8c745249b0786fb3f708b69"
I20260620 23:12:52.243223 18866 raft_consensus.cc:3037] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:52.243994 18586 raft_consensus.cc:2454] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0eb8ca9ef5de4435bde4bec9d837acd2 in term 1.
I20260620 23:12:52.244309 18659 leader_election.cc:302] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eb8ca9ef5de4435bde4bec9d837acd2, a1316ced4e934746b29fba8e79522b9f; no voters:
I20260620 23:12:52.244457 18940 raft_consensus.cc:2781] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:52.244571 18866 raft_consensus.cc:2454] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0eb8ca9ef5de4435bde4bec9d837acd2 in term 1.
I20260620 23:12:52.244640 18940 raft_consensus.cc:686] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 LEADER]: Becoming Leader. State: Replica: 0eb8ca9ef5de4435bde4bec9d837acd2, State: Running, Role: LEADER
I20260620 23:12:52.244796 18940 consensus_queue.cc:237] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.245826 18429 catalog_manager.cc:5495] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2). New cstate: current_term: 1 leader_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } health_report { overall_health: UNKNOWN } } }
W20260620 23:12:52.262800 18913 tablet.cc:2369] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 23:12:52 dist-test-slave-pswb krb5kdc[18384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997171, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:52.305658 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.297450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.16.79.62:46409)
0620 23:12:52.297519 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.297521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.297819 (+ 298us) server_negotiation.cc:407] Connection header received
0620 23:12:52.297866 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.297867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.297888 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.297982 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:52.298795 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.299309 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.300064 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.300247 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.300305 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.302943 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:52.302945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:52.302945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:52.302952 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:52.304530 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:52.305026 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:52.305027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:52.305027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:52.305079 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:52.305257 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:52.305258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:52.305258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:52.305351 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:52.305401 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.305533 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.305594 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:12:52.305833 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.297420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46409 (local address 127.0.0.1:51386)
0620 23:12:52.297678 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.297693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.297760 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.298043 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.298045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.298604 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:52.298665 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.298669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.299393 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.299394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.299975 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.299977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.300319 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.300320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.300363 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.300990 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:52.300995 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:52.302834 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:52.304627 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:52.304634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:52.304635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:52.304957 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:52.305142 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:52.305143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:52.305143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:52.305202 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:52.305408 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:52.305414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:52.305455 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.305516 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.305662 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":55,"thread_start_us":175,"threads_started":1}
I20260620 23:12:52.307865 18429 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:51386:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:12:52.308058 18429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:12:52.309633 18846 tablet_service.cc:1467] Processing CreateTablet for tablet f9386ce248ee4599b055ef62a566769f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3a96b1cad85d4b8a9f36e4aabdc4e9cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:12:52.309640 18706 tablet_service.cc:1467] Processing CreateTablet for tablet f9386ce248ee4599b055ef62a566769f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3a96b1cad85d4b8a9f36e4aabdc4e9cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:12:52.309810 18846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9386ce248ee4599b055ef62a566769f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:52.309834 18706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9386ce248ee4599b055ef62a566769f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:52.311022 18935 tablet_bootstrap.cc:492] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69: Bootstrap starting.
I20260620 23:12:52.311142 18936 tablet_bootstrap.cc:492] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: Bootstrap starting.
I20260620 23:12:52.311120 18548 tablet_service.cc:1467] Processing CreateTablet for tablet f9386ce248ee4599b055ef62a566769f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3a96b1cad85d4b8a9f36e4aabdc4e9cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:12:52.311270 18548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9386ce248ee4599b055ef62a566769f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:52.311610 18936 tablet_bootstrap.cc:654] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:52.311627 18935 tablet_bootstrap.cc:654] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:52.312469 18939 tablet_bootstrap.cc:492] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f: Bootstrap starting.
I20260620 23:12:52.312489 18936 tablet_bootstrap.cc:492] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: No bootstrap required, opened a new log
I20260620 23:12:52.312556 18936 ts_tablet_manager.cc:1397] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:52.312768 18935 tablet_bootstrap.cc:492] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69: No bootstrap required, opened a new log
I20260620 23:12:52.312806 18935 ts_tablet_manager.cc:1397] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:12:52.312768 18936 raft_consensus.cc:348] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.312858 18936 raft_consensus.cc:374] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:52.312887 18936 raft_consensus.cc:729] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb8ca9ef5de4435bde4bec9d837acd2, State: Initialized, Role: FOLLOWER
I20260620 23:12:52.312913 18939 tablet_bootstrap.cc:654] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:52.312934 18936 consensus_queue.cc:260] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.312942 18935 raft_consensus.cc:348] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.313002 18935 raft_consensus.cc:374] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:52.313021 18935 raft_consensus.cc:729] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80a26c5c8c745249b0786fb3f708b69, State: Initialized, Role: FOLLOWER
I20260620 23:12:52.313027 18936 ts_tablet_manager.cc:1428] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:52.313066 18935 consensus_queue.cc:260] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.313071 18940 tablet_replica.cc:439] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } } }
I20260620 23:12:52.313166 18940 tablet_replica.cc:442] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:52.313176 18935 ts_tablet_manager.cc:1428] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:52.313411 18942 tablet_replica.cc:439] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } } }
I20260620 23:12:52.313499 18942 tablet_replica.cc:442] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:52.313639 18939 tablet_bootstrap.cc:492] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f: No bootstrap required, opened a new log
I20260620 23:12:52.313709 18939 ts_tablet_manager.cc:1397] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:52.313865 18939 raft_consensus.cc:348] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.313953 18939 raft_consensus.cc:374] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:52.313987 18939 raft_consensus.cc:729] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1316ced4e934746b29fba8e79522b9f, State: Initialized, Role: FOLLOWER
I20260620 23:12:52.314031 18939 consensus_queue.cc:260] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.314157 18939 ts_tablet_manager.cc:1428] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:12:52.314179 18943 tablet_replica.cc:439] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } } }
I20260620 23:12:52.314286 18943 tablet_replica.cc:442] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:52.411692 18940 raft_consensus.cc:482] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:12:52.411819 18940 raft_consensus.cc:504] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.412089 18940 leader_election.cc:288] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503), a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3:40837)
I20260620 23:12:52.412437 18866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9386ce248ee4599b055ef62a566769f" candidate_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80a26c5c8c745249b0786fb3f708b69" is_pre_election: true
I20260620 23:12:52.412446 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9386ce248ee4599b055ef62a566769f" candidate_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1316ced4e934746b29fba8e79522b9f" is_pre_election: true
I20260620 23:12:52.412546 18586 raft_consensus.cc:2454] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0eb8ca9ef5de4435bde4bec9d837acd2 in term 0.
I20260620 23:12:52.412550 18866 raft_consensus.cc:2454] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0eb8ca9ef5de4435bde4bec9d837acd2 in term 0.
I20260620 23:12:52.412741 18659 leader_election.cc:302] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eb8ca9ef5de4435bde4bec9d837acd2, a1316ced4e934746b29fba8e79522b9f; no voters:
I20260620 23:12:52.412882 18940 raft_consensus.cc:2781] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:12:52.412931 18940 raft_consensus.cc:482] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:12:52.412958 18940 raft_consensus.cc:3037] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:52.413745 18940 raft_consensus.cc:504] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.413883 18940 leader_election.cc:288] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [CANDIDATE]: Term 1 election: Requested vote from peers a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503), a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3:40837)
I20260620 23:12:52.414048 18866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9386ce248ee4599b055ef62a566769f" candidate_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80a26c5c8c745249b0786fb3f708b69"
I20260620 23:12:52.414124 18866 raft_consensus.cc:3037] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:52.414109 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9386ce248ee4599b055ef62a566769f" candidate_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1316ced4e934746b29fba8e79522b9f"
I20260620 23:12:52.414180 18586 raft_consensus.cc:3037] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:52.414929 18866 raft_consensus.cc:2454] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0eb8ca9ef5de4435bde4bec9d837acd2 in term 1.
I20260620 23:12:52.415022 18586 raft_consensus.cc:2454] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0eb8ca9ef5de4435bde4bec9d837acd2 in term 1.
I20260620 23:12:52.415132 18660 leader_election.cc:302] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eb8ca9ef5de4435bde4bec9d837acd2, a80a26c5c8c745249b0786fb3f708b69; no voters:
I20260620 23:12:52.415272 18940 raft_consensus.cc:2781] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:52.415338 18940 raft_consensus.cc:686] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 LEADER]: Becoming Leader. State: Replica: 0eb8ca9ef5de4435bde4bec9d837acd2, State: Running, Role: LEADER
I20260620 23:12:52.415395 18940 consensus_queue.cc:237] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.415647 18945 tablet_replica.cc:439] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: TxnStatusTablet state changed. Reason: New leader 0eb8ca9ef5de4435bde4bec9d837acd2. Latest consensus state: current_term: 1 leader_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } } }
I20260620 23:12:52.415727 18945 tablet_replica.cc:442] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:12:52.415882 18957 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:12:52.416010 18957 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:12:52.415997 18429 catalog_manager.cc:5495] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2). New cstate: current_term: 1 leader_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } health_report { overall_health: UNKNOWN } } }
I20260620 23:12:52.431260 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.428510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34951 (local address 127.0.0.1:40988)
0620 23:12:52.428580 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.428593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.428652 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.428834 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.428835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.428836 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:12:52.428881 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.428882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.429779 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.429780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.430343 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.430344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.430804 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.430805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.430858 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.430871 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:12:52.431137 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:12:52.431140 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.431166 (+ 26us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.431201 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 23:12:52.431275 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.428589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.16.79.2:34951)
0620 23:12:52.428653 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.428655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.428666 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:12:52.428733 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.428734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.428764 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.428834 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:52.428942 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.429684 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.430428 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.430734 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.430804 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.430963 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:52.431076 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:52.431158 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.431182 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.431201 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:12:52.433682 18586 raft_consensus.cc:1261] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Refusing update from remote peer 0eb8ca9ef5de4435bde4bec9d837acd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:52.433687 18866 raft_consensus.cc:1261] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Refusing update from remote peer 0eb8ca9ef5de4435bde4bec9d837acd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:52.434003 18945 consensus_queue.cc:1035] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:52.434121 18940 consensus_queue.cc:1035] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:52.435166 18943 tablet_replica.cc:439] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f: TxnStatusTablet state changed. Reason: New leader 0eb8ca9ef5de4435bde4bec9d837acd2. Latest consensus state: current_term: 1 leader_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } } }
I20260620 23:12:52.435276 18943 tablet_replica.cc:442] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:52.435240 18942 tablet_replica.cc:439] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69: TxnStatusTablet state changed. Reason: New leader 0eb8ca9ef5de4435bde4bec9d837acd2. Latest consensus state: current_term: 1 leader_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } } }
I20260620 23:12:52.435340 18942 tablet_replica.cc:442] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:52.436589 18945 tablet_replica.cc:439] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } } }
I20260620 23:12:52.436666 18940 tablet_replica.cc:439] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } } }
I20260620 23:12:52.436753 18960 mvcc.cc:205] Tried to move back new op lower bound from 7299060418285215744 to 7299060418213752832. Current Snapshot: MvccSnapshot[applied={T|T < 7299060418285215744}]
I20260620 23:12:52.436681 18945 tablet_replica.cc:442] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:12:52.436960 18943 tablet_replica.cc:439] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } } }
I20260620 23:12:52.436769 18940 tablet_replica.cc:442] T f9386ce248ee4599b055ef62a566769f P 0eb8ca9ef5de4435bde4bec9d837acd2: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:12:52.437041 18943 tablet_replica.cc:442] T f9386ce248ee4599b055ef62a566769f P a1316ced4e934746b29fba8e79522b9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:52.437497 18942 tablet_replica.cc:439] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } } }
I20260620 23:12:52.437572 18942 tablet_replica.cc:442] T f9386ce248ee4599b055ef62a566769f P a80a26c5c8c745249b0786fb3f708b69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:12:52.437795 18962 mvcc.cc:205] Tried to move back new op lower bound from 7299060418285215744 to 7299060418213752832. Current Snapshot: MvccSnapshot[applied={T|T < 7299060418285215744 or (T in {7299060418285215744})}]
I20260620 23:12:52.438136 18961 mvcc.cc:205] Tried to move back new op lower bound from 7299060418285215744 to 7299060418213752832. Current Snapshot: MvccSnapshot[applied={T|T < 7299060418285215744 or (T in {7299060418285215744})}]
I20260620 23:12:52.443720 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.440883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.16.79.2:34951)
0620 23:12:52.440954 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.440956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.440966 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:12:52.440989 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.440990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.440999 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.441068 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:52.441279 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.441976 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.442894 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.443197 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.443273 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.443477 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:52.443509 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:52.443567 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.443636 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.443654 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:12:52.449188 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.446348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:43061 (local address 127.16.79.2:34951)
0620 23:12:52.446385 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.446387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.446514 (+ 127us) server_negotiation.cc:407] Connection header received
0620 23:12:52.446538 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.446538 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.446547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.446610 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:52.446695 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.447397 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.448347 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.448632 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.448733 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.448944 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:52.448974 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:52.449027 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.449078 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.449094 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:12:52.449195 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.446290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34951 (local address 127.16.79.2:43061)
0620 23:12:52.446446 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.446454 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.446502 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.446611 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.446611 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.446613 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:12:52.446659 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.446660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.447505 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.447507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.448241 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.448243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.448775 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.448776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.448838 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.448850 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:12:52.449039 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:12:52.449040 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.449074 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.449115 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260620 23:12:52.451083 18866 raft_consensus.cc:1261] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Refusing update from remote peer 0eb8ca9ef5de4435bde4bec9d837acd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:52.451093 18586 raft_consensus.cc:1261] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Refusing update from remote peer 0eb8ca9ef5de4435bde4bec9d837acd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:52.451293 18945 consensus_queue.cc:1035] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:52.451387 18940 consensus_queue.cc:1035] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:52.462384 18429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:52.462505 18429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:52.463482 18429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594: Sending DeleteTablet for 3 replicas of tablet f0f39b9095e34fa5ab0a0b2eee9e09dd
I20260620 23:12:52.463878 18548 tablet_service.cc:1514] Processing DeleteTablet for tablet f0f39b9095e34fa5ab0a0b2eee9e09dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:52 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:57760
I20260620 23:12:52.463969 18846 tablet_service.cc:1514] Processing DeleteTablet for tablet f0f39b9095e34fa5ab0a0b2eee9e09dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:52 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:44060
I20260620 23:12:52.464200 18705 tablet_service.cc:1514] Processing DeleteTablet for tablet f0f39b9095e34fa5ab0a0b2eee9e09dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:52 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:40976
I20260620 23:12:52.464344 18976 tablet_replica.cc:330] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f: stopping tablet replica
I20260620 23:12:52.464437 18976 raft_consensus.cc:2229] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:52.464464 18978 tablet_replica.cc:330] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2: stopping tablet replica
I20260620 23:12:52.464495 18976 raft_consensus.cc:2258] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:52.464740 18978 raft_consensus.cc:2229] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:12:52.464881 18978 raft_consensus.cc:2258] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:52.464838 18429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 23:12:52.464972 18976 ts_tablet_manager.cc:1905] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 23:12:52.465013 18429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:12:52.465411 18978 ts_tablet_manager.cc:1905] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:52.465808 18977 tablet_replica.cc:330] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69: stopping tablet replica
I20260620 23:12:52.465988 18977 raft_consensus.cc:2229] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:52.466120 18977 raft_consensus.cc:2258] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:52.466675 18977 ts_tablet_manager.cc:1905] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:52.466902 18548 tablet_service.cc:1467] Processing CreateTablet for tablet 783ba38892c3427c8e0b7a93316fef39 (DEFAULT_TABLE table=test-table [id=cc0c715f5dae416caff6c5f7a30c4668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:52.467042 18846 tablet_service.cc:1467] Processing CreateTablet for tablet 783ba38892c3427c8e0b7a93316fef39 (DEFAULT_TABLE table=test-table [id=cc0c715f5dae416caff6c5f7a30c4668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:52.467185 18846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783ba38892c3427c8e0b7a93316fef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:52.468858 18705 tablet_service.cc:1467] Processing CreateTablet for tablet 783ba38892c3427c8e0b7a93316fef39 (DEFAULT_TABLE table=test-table [id=cc0c715f5dae416caff6c5f7a30c4668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:52.468998 18705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783ba38892c3427c8e0b7a93316fef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:52.469165 18978 ts_tablet_manager.cc:1918] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:12:52.469336 18978 log.cc:1196] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/f0f39b9095e34fa5ab0a0b2eee9e09dd
I20260620 23:12:52.469573 18977 ts_tablet_manager.cc:1918] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:12:52.469699 18978 ts_tablet_manager.cc:1939] T f0f39b9095e34fa5ab0a0b2eee9e09dd P 0eb8ca9ef5de4435bde4bec9d837acd2: Deleting consensus metadata
I20260620 23:12:52.469717 18977 log.cc:1196] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/f0f39b9095e34fa5ab0a0b2eee9e09dd
I20260620 23:12:52.470091 18977 ts_tablet_manager.cc:1939] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a80a26c5c8c745249b0786fb3f708b69: Deleting consensus metadata
I20260620 23:12:52.470641 18936 tablet_bootstrap.cc:492] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2: Bootstrap starting.
I20260620 23:12:52.470639 18935 tablet_bootstrap.cc:492] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69: Bootstrap starting.
I20260620 23:12:52.471091 18936 tablet_bootstrap.cc:654] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:52.471086 18935 tablet_bootstrap.cc:654] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:52.471590 18415 catalog_manager.cc:4841] TS 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2:34951): tablet f0f39b9095e34fa5ab0a0b2eee9e09dd (table test-table [id=84d103106b704fe2ab61abe19ad0dbbb]) successfully deleted
I20260620 23:12:52.471688 18416 catalog_manager.cc:4841] TS a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3:40837): tablet f0f39b9095e34fa5ab0a0b2eee9e09dd (table test-table [id=84d103106b704fe2ab61abe19ad0dbbb]) successfully deleted
I20260620 23:12:52.471894 18548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783ba38892c3427c8e0b7a93316fef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:52.472218 18935 tablet_bootstrap.cc:492] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69: No bootstrap required, opened a new log
I20260620 23:12:52.472291 18935 ts_tablet_manager.cc:1397] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:12:52.472492 18935 raft_consensus.cc:348] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.472604 18935 raft_consensus.cc:374] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:52.472632 18935 raft_consensus.cc:729] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80a26c5c8c745249b0786fb3f708b69, State: Initialized, Role: FOLLOWER
I20260620 23:12:52.472667 18935 consensus_queue.cc:260] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.472788 18935 ts_tablet_manager.cc:1428] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:12:52.473088 18939 tablet_bootstrap.cc:492] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f: Bootstrap starting.
I20260620 23:12:52.473477 18939 tablet_bootstrap.cc:654] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:52.473861 18976 ts_tablet_manager.cc:1918] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:12:52.473923 18976 log.cc:1196] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/wal/wals/f0f39b9095e34fa5ab0a0b2eee9e09dd
I20260620 23:12:52.474189 18976 ts_tablet_manager.cc:1939] T f0f39b9095e34fa5ab0a0b2eee9e09dd P a1316ced4e934746b29fba8e79522b9f: Deleting consensus metadata
I20260620 23:12:52.474661 18415 catalog_manager.cc:4841] TS a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503): tablet f0f39b9095e34fa5ab0a0b2eee9e09dd (table test-table [id=84d103106b704fe2ab61abe19ad0dbbb]) successfully deleted
I20260620 23:12:52.474881 18939 tablet_bootstrap.cc:492] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f: No bootstrap required, opened a new log
I20260620 23:12:52.474913 18939 ts_tablet_manager.cc:1397] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:12:52.475028 18939 raft_consensus.cc:348] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.475069 18939 raft_consensus.cc:374] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:52.475082 18939 raft_consensus.cc:729] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1316ced4e934746b29fba8e79522b9f, State: Initialized, Role: FOLLOWER
I20260620 23:12:52.475142 18939 consensus_queue.cc:260] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.475255 18939 ts_tablet_manager.cc:1428] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:52.477066 18936 tablet_bootstrap.cc:492] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2: No bootstrap required, opened a new log
I20260620 23:12:52.477135 18936 ts_tablet_manager.cc:1397] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 23:12:52.477295 18936 raft_consensus.cc:348] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.477370 18936 raft_consensus.cc:374] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:52.477398 18936 raft_consensus.cc:729] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb8ca9ef5de4435bde4bec9d837acd2, State: Initialized, Role: FOLLOWER
I20260620 23:12:52.477439 18936 consensus_queue.cc:260] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.477528 18936 ts_tablet_manager.cc:1428] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:12:52.581672 18942 raft_consensus.cc:482] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:12:52.581827 18942 raft_consensus.cc:504] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.582170 18942 leader_election.cc:288] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503), 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2:34951)
I20260620 23:12:52.586254 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.582407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:34503 (local address 127.16.79.3:39773)
0620 23:12:52.582634 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.582647 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.582762 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.582976 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.582977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.582981 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:52.583036 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.583038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.584027 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.584028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.585045 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.585047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.585913 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.585914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.586000 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.586006 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.586056 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.586114 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":46,"thread_start_us":75,"threads_started":1}
I20260620 23:12:52.586231 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.582512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:39773 (local address 127.16.79.1:34503)
0620 23:12:52.582615 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.582618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.582725 (+ 107us) server_negotiation.cc:407] Connection header received
0620 23:12:52.582851 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.582852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.582864 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.582898 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:52.583121 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.583923 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.585127 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.585728 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.585899 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.585916 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.586077 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.586134 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:12:52.586753 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.582594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34951 (local address 127.16.79.3:45007)
0620 23:12:52.582756 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.582770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.582926 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.583143 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.583144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.583147 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:52.583239 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.583242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.584108 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.584109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.585399 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.585402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.586443 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.586444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.586541 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.586544 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.586617 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.586668 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":35,"threads_started":1}
I20260620 23:12:52.586762 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.582726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:45007 (local address 127.16.79.2:34951)
0620 23:12:52.582771 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.582772 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.582882 (+ 110us) server_negotiation.cc:407] Connection header received
0620 23:12:52.583023 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.583024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.583037 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.583069 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:52.583340 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.584006 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.585589 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.586257 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.586345 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.586360 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.586641 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.586688 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:12:52.587090 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783ba38892c3427c8e0b7a93316fef39" candidate_uuid: "a80a26c5c8c745249b0786fb3f708b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" is_pre_election: true
I20260620 23:12:52.587111 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783ba38892c3427c8e0b7a93316fef39" candidate_uuid: "a80a26c5c8c745249b0786fb3f708b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1316ced4e934746b29fba8e79522b9f" is_pre_election: true
I20260620 23:12:52.587245 18586 raft_consensus.cc:2454] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80a26c5c8c745249b0786fb3f708b69 in term 0.
I20260620 23:12:52.587270 18726 raft_consensus.cc:2454] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80a26c5c8c745249b0786fb3f708b69 in term 0.
I20260620 23:12:52.587492 18799 leader_election.cc:302] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eb8ca9ef5de4435bde4bec9d837acd2, a80a26c5c8c745249b0786fb3f708b69; no voters:
I20260620 23:12:52.587710 18942 raft_consensus.cc:2781] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:12:52.587762 18942 raft_consensus.cc:482] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:12:52.587895 18942 raft_consensus.cc:3037] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:52.588627 18942 raft_consensus.cc:504] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.588810 18942 leader_election.cc:288] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [CANDIDATE]: Term 1 election: Requested vote from peers a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503), 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2:34951)
I20260620 23:12:52.589013 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783ba38892c3427c8e0b7a93316fef39" candidate_uuid: "a80a26c5c8c745249b0786fb3f708b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2"
I20260620 23:12:52.589007 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783ba38892c3427c8e0b7a93316fef39" candidate_uuid: "a80a26c5c8c745249b0786fb3f708b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1316ced4e934746b29fba8e79522b9f"
I20260620 23:12:52.589099 18586 raft_consensus.cc:3037] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:52.589099 18726 raft_consensus.cc:3037] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:52.589810 18586 raft_consensus.cc:2454] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80a26c5c8c745249b0786fb3f708b69 in term 1.
I20260620 23:12:52.589874 18726 raft_consensus.cc:2454] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80a26c5c8c745249b0786fb3f708b69 in term 1.
I20260620 23:12:52.590000 18799 leader_election.cc:302] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eb8ca9ef5de4435bde4bec9d837acd2, a80a26c5c8c745249b0786fb3f708b69; no voters:
I20260620 23:12:52.590112 18942 raft_consensus.cc:2781] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:52.590229 18942 raft_consensus.cc:686] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 1 LEADER]: Becoming Leader. State: Replica: a80a26c5c8c745249b0786fb3f708b69, State: Running, Role: LEADER
I20260620 23:12:52.590304 18942 consensus_queue.cc:237] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:52.591063 18429 catalog_manager.cc:5495] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3). New cstate: current_term: 1 leader_uuid: "a80a26c5c8c745249b0786fb3f708b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } health_report { overall_health: HEALTHY } } }
I20260620 23:12:52.618996 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.615885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.16.79.3:40837)
0620 23:12:52.615930 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.615932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.616034 (+ 102us) server_negotiation.cc:407] Connection header received
0620 23:12:52.616089 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.616090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.616119 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.616304 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:52.616480 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.617021 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.618181 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.618374 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.618439 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.618639 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:52.618742 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:52.618807 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.618896 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.618919 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:12:52.625103 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.621993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:56769 (local address 127.16.79.2:34951)
0620 23:12:52.622071 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.622073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.622089 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:12:52.622124 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.622125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.622136 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.622217 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:52.622404 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.623185 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.624300 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.624480 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.624550 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.624827 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:52.624858 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:52.624918 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.625033 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.625051 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:12:52.625137 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.621846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34951 (local address 127.16.79.3:56769)
0620 23:12:52.622011 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.622021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.622068 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.622245 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.622246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.622248 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:12:52.622297 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.622299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.623287 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.623288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.624197 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.624200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.624581 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.624582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.624656 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.624670 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:12:52.624946 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:12:52.624949 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.625003 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.625048 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260620 23:12:52.627226 18586 raft_consensus.cc:1261] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Refusing update from remote peer a80a26c5c8c745249b0786fb3f708b69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:52.627266 18726 raft_consensus.cc:1261] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Refusing update from remote peer a80a26c5c8c745249b0786fb3f708b69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:52.627499 18984 consensus_queue.cc:1035] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:52.627624 18942 consensus_queue.cc:1035] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:52.635478 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.632919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:40837 (local address 127.16.79.2:49169)
0620 23:12:52.632963 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:52.632972 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:52.633006 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:52.633186 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:52.633187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:52.633188 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:12:52.633223 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.633225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.633849 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.633850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.634704 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:52.634707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.635154 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.635155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:52.635197 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.635203 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:12:52.635366 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:12:52.635368 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:12:52.635400 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 23:12:52.635430 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 23:12:52.635484 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:52.632944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:49169 (local address 127.16.79.3:40837)
0620 23:12:52.633069 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:52.633071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:52.633077 (+ 6us) server_negotiation.cc:407] Connection header received
0620 23:12:52.633093 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:52.633094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:52.633104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:52.633184 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:52.633296 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.633762 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.634803 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:52.635071 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:52.635148 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:52.635276 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:52.635307 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:52.635362 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:52.635406 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 23:12:52.635424 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260620 23:12:52.635780 18962 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:12:54.159607 18428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:54.159696 18428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:54.160477 18428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594: Sending DeleteTablet for 3 replicas of tablet 783ba38892c3427c8e0b7a93316fef39
I20260620 23:12:54.160737 18846 tablet_service.cc:1514] Processing DeleteTablet for tablet 783ba38892c3427c8e0b7a93316fef39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:54 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:44060
I20260620 23:12:54.160759 18705 tablet_service.cc:1514] Processing DeleteTablet for tablet 783ba38892c3427c8e0b7a93316fef39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:54 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:40976
I20260620 23:12:54.160758 18548 tablet_service.cc:1514] Processing DeleteTablet for tablet 783ba38892c3427c8e0b7a93316fef39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:54 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:57760
I20260620 23:12:54.160946 19015 tablet_replica.cc:330] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69: stopping tablet replica
I20260620 23:12:54.161052 19017 tablet_replica.cc:330] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2: stopping tablet replica
I20260620 23:12:54.161124 19017 raft_consensus.cc:2229] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:54.161187 19015 raft_consensus.cc:2229] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:12:54.161262 19017 raft_consensus.cc:2258] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:54.161361 19015 raft_consensus.cc:2258] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:54.161615 19017 ts_tablet_manager.cc:1905] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:54.161836 19015 ts_tablet_manager.cc:1905] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:54.162559 19016 tablet_replica.cc:330] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f: stopping tablet replica
I20260620 23:12:54.162683 19016 raft_consensus.cc:2229] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:54.162783 19016 raft_consensus.cc:2258] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:54.163046 19016 ts_tablet_manager.cc:1905] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:54.163689 19017 ts_tablet_manager.cc:1918] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:12:54.163769 19017 log.cc:1196] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/783ba38892c3427c8e0b7a93316fef39
I20260620 23:12:54.164013 19017 ts_tablet_manager.cc:1939] T 783ba38892c3427c8e0b7a93316fef39 P 0eb8ca9ef5de4435bde4bec9d837acd2: Deleting consensus metadata
I20260620 23:12:54.164326 18415 catalog_manager.cc:4841] TS 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2:34951): tablet 783ba38892c3427c8e0b7a93316fef39 (table test-table [id=cc0c715f5dae416caff6c5f7a30c4668]) successfully deleted
I20260620 23:12:54.164731 19016 ts_tablet_manager.cc:1918] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:12:54.164798 19016 log.cc:1196] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/wal/wals/783ba38892c3427c8e0b7a93316fef39
I20260620 23:12:54.165033 19016 ts_tablet_manager.cc:1939] T 783ba38892c3427c8e0b7a93316fef39 P a1316ced4e934746b29fba8e79522b9f: Deleting consensus metadata
I20260620 23:12:54.165140 19015 ts_tablet_manager.cc:1918] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:12:54.165194 19015 log.cc:1196] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/783ba38892c3427c8e0b7a93316fef39
I20260620 23:12:54.165330 18415 catalog_manager.cc:4841] TS a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503): tablet 783ba38892c3427c8e0b7a93316fef39 (table test-table [id=cc0c715f5dae416caff6c5f7a30c4668]) successfully deleted
I20260620 23:12:54.165438 19015 ts_tablet_manager.cc:1939] T 783ba38892c3427c8e0b7a93316fef39 P a80a26c5c8c745249b0786fb3f708b69: Deleting consensus metadata
I20260620 23:12:54.165736 18416 catalog_manager.cc:4841] TS a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3:40837): tablet 783ba38892c3427c8e0b7a93316fef39 (table test-table [id=cc0c715f5dae416caff6c5f7a30c4668]) successfully deleted
Jun 20 23:12:54 dist-test-slave-pswb krb5kdc[18384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:12:54 dist-test-slave-pswb krb5kdc[18384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:54.197130 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:54.179053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.16.79.62:46409)
0620 23:12:54.179182 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:54.179184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:54.179191 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:12:54.179211 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:54.179212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:54.179223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:54.179284 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:54.179782 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.180452 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.181330 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.181579 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.181649 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:54.194547 (+ 12898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:54.194549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:54.194549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:54.194556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:54.196043 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:54.196407 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:54.196407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:54.196408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:54.196461 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:54.196634 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:54.196634 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:54.196634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:54.196806 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:54.196884 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:54.196998 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 23:12:54.197061 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":82,"threads_started":1}
I20260620 23:12:54.198915 18428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:12:54.199069 18428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:12:54.201380 18705 tablet_service.cc:1467] Processing CreateTablet for tablet ef288253c4354ccea79d14da6cfa74f2 (DEFAULT_TABLE table=test-table [id=422b745ebbcc4bd09ecb7921e8915e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:54.201413 18548 tablet_service.cc:1467] Processing CreateTablet for tablet ef288253c4354ccea79d14da6cfa74f2 (DEFAULT_TABLE table=test-table [id=422b745ebbcc4bd09ecb7921e8915e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:54.201413 18846 tablet_service.cc:1467] Processing CreateTablet for tablet ef288253c4354ccea79d14da6cfa74f2 (DEFAULT_TABLE table=test-table [id=422b745ebbcc4bd09ecb7921e8915e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:54.201534 18705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef288253c4354ccea79d14da6cfa74f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:54.201592 18846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef288253c4354ccea79d14da6cfa74f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:54.201592 18548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef288253c4354ccea79d14da6cfa74f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:54.203006 19026 tablet_bootstrap.cc:492] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Bootstrap starting.
I20260620 23:12:54.203303 19028 tablet_bootstrap.cc:492] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f: Bootstrap starting.
I20260620 23:12:54.203384 19026 tablet_bootstrap.cc:654] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:54.203451 19027 tablet_bootstrap.cc:492] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69: Bootstrap starting.
I20260620 23:12:54.203707 19028 tablet_bootstrap.cc:654] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:54.203923 19027 tablet_bootstrap.cc:654] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:54.204125 19026 tablet_bootstrap.cc:492] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2: No bootstrap required, opened a new log
I20260620 23:12:54.204308 19026 ts_tablet_manager.cc:1397] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:54.204485 19028 tablet_bootstrap.cc:492] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f: No bootstrap required, opened a new log
I20260620 23:12:54.204450 19026 raft_consensus.cc:348] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.204528 19026 raft_consensus.cc:374] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:54.204599 19026 raft_consensus.cc:729] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb8ca9ef5de4435bde4bec9d837acd2, State: Initialized, Role: FOLLOWER
I20260620 23:12:54.204604 19027 tablet_bootstrap.cc:492] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69: No bootstrap required, opened a new log
I20260620 23:12:54.204663 19027 ts_tablet_manager.cc:1397] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:54.204658 19026 consensus_queue.cc:260] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.204668 19028 ts_tablet_manager.cc:1397] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:54.204810 19026 ts_tablet_manager.cc:1428] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:54.204861 19028 raft_consensus.cc:348] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.204943 19028 raft_consensus.cc:374] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:54.204967 19028 raft_consensus.cc:729] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1316ced4e934746b29fba8e79522b9f, State: Initialized, Role: FOLLOWER
I20260620 23:12:54.205034 19028 consensus_queue.cc:260] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.205199 19028 ts_tablet_manager.cc:1428] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:54.205042 19027 raft_consensus.cc:348] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.205281 19027 raft_consensus.cc:374] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:54.205446 19027 raft_consensus.cc:729] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80a26c5c8c745249b0786fb3f708b69, State: Initialized, Role: FOLLOWER
I20260620 23:12:54.205528 19027 consensus_queue.cc:260] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.205740 19027 ts_tablet_manager.cc:1428] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 23:12:54.349107 19009 raft_consensus.cc:482] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:12:54.349198 19009 raft_consensus.cc:504] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.349366 19009 leader_election.cc:288] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503), 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2:34951)
I20260620 23:12:54.349558 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef288253c4354ccea79d14da6cfa74f2" candidate_uuid: "a80a26c5c8c745249b0786fb3f708b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" is_pre_election: true
I20260620 23:12:54.349558 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef288253c4354ccea79d14da6cfa74f2" candidate_uuid: "a80a26c5c8c745249b0786fb3f708b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1316ced4e934746b29fba8e79522b9f" is_pre_election: true
I20260620 23:12:54.349665 18726 raft_consensus.cc:2454] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80a26c5c8c745249b0786fb3f708b69 in term 0.
I20260620 23:12:54.349665 18586 raft_consensus.cc:2454] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80a26c5c8c745249b0786fb3f708b69 in term 0.
I20260620 23:12:54.349839 18799 leader_election.cc:302] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1316ced4e934746b29fba8e79522b9f, a80a26c5c8c745249b0786fb3f708b69; no voters:
I20260620 23:12:54.349951 19009 raft_consensus.cc:2781] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:12:54.349992 19009 raft_consensus.cc:482] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:12:54.350016 19009 raft_consensus.cc:3037] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:54.350726 19009 raft_consensus.cc:504] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.350848 19009 leader_election.cc:288] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [CANDIDATE]: Term 1 election: Requested vote from peers a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503), 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2:34951)
I20260620 23:12:54.351022 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef288253c4354ccea79d14da6cfa74f2" candidate_uuid: "a80a26c5c8c745249b0786fb3f708b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2"
I20260620 23:12:54.351027 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef288253c4354ccea79d14da6cfa74f2" candidate_uuid: "a80a26c5c8c745249b0786fb3f708b69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1316ced4e934746b29fba8e79522b9f"
I20260620 23:12:54.351087 18586 raft_consensus.cc:3037] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:54.351087 18726 raft_consensus.cc:3037] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:54.351830 18726 raft_consensus.cc:2454] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80a26c5c8c745249b0786fb3f708b69 in term 1.
I20260620 23:12:54.351823 18586 raft_consensus.cc:2454] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80a26c5c8c745249b0786fb3f708b69 in term 1.
I20260620 23:12:54.351987 18799 leader_election.cc:302] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1316ced4e934746b29fba8e79522b9f, a80a26c5c8c745249b0786fb3f708b69; no voters:
I20260620 23:12:54.352093 19009 raft_consensus.cc:2781] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:54.352149 19009 raft_consensus.cc:686] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 1 LEADER]: Becoming Leader. State: Replica: a80a26c5c8c745249b0786fb3f708b69, State: Running, Role: LEADER
I20260620 23:12:54.352200 19009 consensus_queue.cc:237] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.352849 18428 catalog_manager.cc:5495] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 reported cstate change: term changed from 0 to 1, leader changed from <none> to a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3). New cstate: current_term: 1 leader_uuid: "a80a26c5c8c745249b0786fb3f708b69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } health_report { overall_health: HEALTHY } } }
I20260620 23:12:54.390789 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:54.388079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.16.79.3:40837)
0620 23:12:54.388276 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:54.388278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:54.388284 (+ 6us) server_negotiation.cc:407] Connection header received
0620 23:12:54.388303 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:54.388304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:54.388315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:54.388398 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:54.388567 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.389224 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.390069 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.390279 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.390354 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:54.390510 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:54.390544 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:54.390610 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:54.390665 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 23:12:54.390689 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260620 23:12:54.392968 18586 raft_consensus.cc:1261] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Refusing update from remote peer a80a26c5c8c745249b0786fb3f708b69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:54.392989 18726 raft_consensus.cc:1261] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Refusing update from remote peer a80a26c5c8c745249b0786fb3f708b69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:54.393194 18993 consensus_queue.cc:1035] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:54.393301 19009 consensus_queue.cc:1035] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:54.402733 18428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:54.402833 18428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:54.403669 18428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594: Sending DeleteTablet for 3 replicas of tablet ef288253c4354ccea79d14da6cfa74f2
I20260620 23:12:54.403927 18548 tablet_service.cc:1514] Processing DeleteTablet for tablet ef288253c4354ccea79d14da6cfa74f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:54 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:57760
I20260620 23:12:54.403947 18705 tablet_service.cc:1514] Processing DeleteTablet for tablet ef288253c4354ccea79d14da6cfa74f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:54 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:40976
I20260620 23:12:54.404029 19016 tablet_replica.cc:330] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f: stopping tablet replica
I20260620 23:12:54.404083 19016 raft_consensus.cc:2229] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:54.404069 19017 tablet_replica.cc:330] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2: stopping tablet replica
I20260620 23:12:54.404127 19016 raft_consensus.cc:2258] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:54.404188 19017 raft_consensus.cc:2229] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:54.404259 19017 raft_consensus.cc:2258] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:54.404260 18846 tablet_service.cc:1514] Processing DeleteTablet for tablet ef288253c4354ccea79d14da6cfa74f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:54 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:44060
I20260620 23:12:54.404415 19015 tablet_replica.cc:330] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69: stopping tablet replica
I20260620 23:12:54.404486 19015 raft_consensus.cc:2229] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:12:54.404505 19016 ts_tablet_manager.cc:1905] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:54.404549 19015 raft_consensus.cc:2258] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:54.404438 18428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:12:54.404675 18428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:12:54.404882 19015 ts_tablet_manager.cc:1905] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:54.405218 19017 ts_tablet_manager.cc:1905] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:54.406297 19016 ts_tablet_manager.cc:1918] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:12:54.406355 19016 log.cc:1196] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-0/wal/wals/ef288253c4354ccea79d14da6cfa74f2
I20260620 23:12:54.406414 18548 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff9ffe5e2b949018aefbc1fb8d776e2 (DEFAULT_TABLE table=test-table [id=7c2959ed70794425baa2c9d185504fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:54.406536 18548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff9ffe5e2b949018aefbc1fb8d776e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:54.406575 19016 ts_tablet_manager.cc:1939] T ef288253c4354ccea79d14da6cfa74f2 P a1316ced4e934746b29fba8e79522b9f: Deleting consensus metadata
I20260620 23:12:54.406677 19015 ts_tablet_manager.cc:1918] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:12:54.406726 19015 log.cc:1196] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/ef288253c4354ccea79d14da6cfa74f2
I20260620 23:12:54.406864 19017 ts_tablet_manager.cc:1918] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:12:54.406916 19017 log.cc:1196] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/ef288253c4354ccea79d14da6cfa74f2
I20260620 23:12:54.406965 19015 ts_tablet_manager.cc:1939] T ef288253c4354ccea79d14da6cfa74f2 P a80a26c5c8c745249b0786fb3f708b69: Deleting consensus metadata
I20260620 23:12:54.407159 19017 ts_tablet_manager.cc:1939] T ef288253c4354ccea79d14da6cfa74f2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Deleting consensus metadata
I20260620 23:12:54.407806 19028 tablet_bootstrap.cc:492] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f: Bootstrap starting.
I20260620 23:12:54.407862 18415 catalog_manager.cc:4841] TS a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503): tablet ef288253c4354ccea79d14da6cfa74f2 (table test-table [id=422b745ebbcc4bd09ecb7921e8915e6c]) successfully deleted
I20260620 23:12:54.407934 18415 catalog_manager.cc:4841] TS 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2:34951): tablet ef288253c4354ccea79d14da6cfa74f2 (table test-table [id=422b745ebbcc4bd09ecb7921e8915e6c]) successfully deleted
I20260620 23:12:54.408046 18416 catalog_manager.cc:4841] TS a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3:40837): tablet ef288253c4354ccea79d14da6cfa74f2 (table test-table [id=422b745ebbcc4bd09ecb7921e8915e6c]) successfully deleted
I20260620 23:12:54.408054 18705 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff9ffe5e2b949018aefbc1fb8d776e2 (DEFAULT_TABLE table=test-table [id=7c2959ed70794425baa2c9d185504fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:54.408160 18705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff9ffe5e2b949018aefbc1fb8d776e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:54.408438 19028 tablet_bootstrap.cc:654] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:54.408776 18846 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff9ffe5e2b949018aefbc1fb8d776e2 (DEFAULT_TABLE table=test-table [id=7c2959ed70794425baa2c9d185504fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:54.408905 18846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff9ffe5e2b949018aefbc1fb8d776e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:54.409121 19028 tablet_bootstrap.cc:492] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f: No bootstrap required, opened a new log
I20260620 23:12:54.409206 19028 ts_tablet_manager.cc:1397] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:54.409421 19028 raft_consensus.cc:348] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.409556 19028 raft_consensus.cc:374] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:54.409567 19026 tablet_bootstrap.cc:492] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Bootstrap starting.
I20260620 23:12:54.409638 19028 raft_consensus.cc:729] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1316ced4e934746b29fba8e79522b9f, State: Initialized, Role: FOLLOWER
I20260620 23:12:54.409735 19028 consensus_queue.cc:260] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.409850 19026 tablet_bootstrap.cc:654] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:54.409874 19027 tablet_bootstrap.cc:492] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69: Bootstrap starting.
I20260620 23:12:54.410015 19028 ts_tablet_manager.cc:1428] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 23:12:54.410241 19027 tablet_bootstrap.cc:654] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:54.410494 19026 tablet_bootstrap.cc:492] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2: No bootstrap required, opened a new log
I20260620 23:12:54.410562 19026 ts_tablet_manager.cc:1397] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:54.410678 19026 raft_consensus.cc:348] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.410758 19026 raft_consensus.cc:374] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:54.410787 19026 raft_consensus.cc:729] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb8ca9ef5de4435bde4bec9d837acd2, State: Initialized, Role: FOLLOWER
I20260620 23:12:54.410841 19026 consensus_queue.cc:260] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.410885 19027 tablet_bootstrap.cc:492] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69: No bootstrap required, opened a new log
I20260620 23:12:54.410928 19027 ts_tablet_manager.cc:1397] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:54.410949 19026 ts_tablet_manager.cc:1428] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:12:54.411051 19027 raft_consensus.cc:348] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.411129 19027 raft_consensus.cc:374] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:54.411152 19027 raft_consensus.cc:729] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80a26c5c8c745249b0786fb3f708b69, State: Initialized, Role: FOLLOWER
I20260620 23:12:54.411206 19027 consensus_queue.cc:260] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.411310 19027 ts_tablet_manager.cc:1428] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:54.440873 19032 raft_consensus.cc:482] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:12:54.440943 19032 raft_consensus.cc:504] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.441221 19032 leader_election.cc:288] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2:34951), a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3:40837)
I20260620 23:12:54.445669 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:54.441538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:49831 (local address 127.16.79.3:40837)
0620 23:12:54.441584 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:54.441586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:54.441897 (+ 311us) server_negotiation.cc:407] Connection header received
0620 23:12:54.441927 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:54.441928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:54.441936 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:54.441991 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:54.442155 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.442945 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.444542 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.445124 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.445202 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:54.445213 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:54.445567 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 23:12:54.445603 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:12:54.445672 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:54.441538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:37207 (local address 127.16.79.2:34951)
0620 23:12:54.441749 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:54.441751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:54.441776 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:12:54.442046 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:54.442047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:54.442055 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:54.442167 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:54.442286 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.442971 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.444542 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.445124 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.445202 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:54.445222 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:54.445567 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 23:12:54.445604 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":60,"threads_started":1}
I20260620 23:12:54.445959 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:54.441484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:40837 (local address 127.16.79.1:49831)
0620 23:12:54.441775 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:54.441787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:54.441851 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:54.442002 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:54.442002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:54.442004 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:54.442064 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:54.442066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.443044 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.443045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:54.444427 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:54.444429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.445270 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.445271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:54.445441 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:54.445452 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:12:54.445553 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:12:54.445634 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":105,"thread_start_us":115,"threads_started":1}
I20260620 23:12:54.446349 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:54.441484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34951 (local address 127.16.79.1:37207)
0620 23:12:54.441688 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:54.441706 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:54.441793 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:54.442171 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:54.442172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:54.442174 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:54.442224 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:54.442225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.443073 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.443074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:54.444427 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:54.444429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.445269 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.445271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:54.445441 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:54.445454 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:12:54.445553 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:12:54.445634 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":106,"thread_start_us":133,"threads_started":1}
I20260620 23:12:54.446369 18866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff9ffe5e2b949018aefbc1fb8d776e2" candidate_uuid: "a1316ced4e934746b29fba8e79522b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80a26c5c8c745249b0786fb3f708b69" is_pre_election: true
I20260620 23:12:54.446473 18866 raft_consensus.cc:2454] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1316ced4e934746b29fba8e79522b9f in term 0.
I20260620 23:12:54.446683 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff9ffe5e2b949018aefbc1fb8d776e2" candidate_uuid: "a1316ced4e934746b29fba8e79522b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" is_pre_election: true
I20260620 23:12:54.446776 18726 raft_consensus.cc:2454] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1316ced4e934746b29fba8e79522b9f in term 0.
I20260620 23:12:54.446923 18520 leader_election.cc:302] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1316ced4e934746b29fba8e79522b9f, a80a26c5c8c745249b0786fb3f708b69; no voters:
I20260620 23:12:54.447084 19032 raft_consensus.cc:2781] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:12:54.447139 19032 raft_consensus.cc:482] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:12:54.447156 19032 raft_consensus.cc:3037] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:54.448081 19032 raft_consensus.cc:504] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.448257 19032 leader_election.cc:288] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [CANDIDATE]: Term 1 election: Requested vote from peers 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2:34951), a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3:40837)
I20260620 23:12:54.448403 18866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff9ffe5e2b949018aefbc1fb8d776e2" candidate_uuid: "a1316ced4e934746b29fba8e79522b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80a26c5c8c745249b0786fb3f708b69"
I20260620 23:12:54.448390 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff9ffe5e2b949018aefbc1fb8d776e2" candidate_uuid: "a1316ced4e934746b29fba8e79522b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2"
I20260620 23:12:54.448457 18866 raft_consensus.cc:3037] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:54.448460 18726 raft_consensus.cc:3037] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:54.449084 18866 raft_consensus.cc:2454] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1316ced4e934746b29fba8e79522b9f in term 1.
I20260620 23:12:54.449132 18726 raft_consensus.cc:2454] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1316ced4e934746b29fba8e79522b9f in term 1.
I20260620 23:12:54.449275 18520 leader_election.cc:302] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1316ced4e934746b29fba8e79522b9f, a80a26c5c8c745249b0786fb3f708b69; no voters:
I20260620 23:12:54.449366 19032 raft_consensus.cc:2781] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:54.449525 19032 raft_consensus.cc:686] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [term 1 LEADER]: Becoming Leader. State: Replica: a1316ced4e934746b29fba8e79522b9f, State: Running, Role: LEADER
I20260620 23:12:54.449579 19032 consensus_queue.cc:237] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } }
I20260620 23:12:54.450273 18428 catalog_manager.cc:5495] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f reported cstate change: term changed from 0 to 1, leader changed from <none> to a1316ced4e934746b29fba8e79522b9f (127.16.79.1). New cstate: current_term: 1 leader_uuid: "a1316ced4e934746b29fba8e79522b9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1316ced4e934746b29fba8e79522b9f" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 } health_report { overall_health: UNKNOWN } } }
I20260620 23:12:54.461069 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:54.458682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57768 (local address 127.16.79.1:34503)
0620 23:12:54.458818 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:54.458820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:54.458829 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:12:54.458861 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:54.458862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:54.458880 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:54.458973 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:54.459070 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.459547 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.460269 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.460490 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.460564 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:54.460715 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:54.460829 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:54.460908 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:54.460963 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 23:12:54.460980 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260620 23:12:54.465255 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:54.463239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:53047 (local address 127.16.79.2:34951)
0620 23:12:54.463275 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:54.463277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:54.463286 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:12:54.463355 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:54.463355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:54.463364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:54.463401 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:54.463489 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.463916 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.464606 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.464877 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.464933 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:54.465092 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:54.465115 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:54.465152 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:54.465196 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 23:12:54.465210 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:12:54.465337 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:54.463107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34951 (local address 127.16.79.1:53047)
0620 23:12:54.463240 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:54.463250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:54.463297 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:54.463401 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:54.463402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:54.463403 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:12:54.463442 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:54.463443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.463992 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.463993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:54.464538 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:54.464540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.464962 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.464963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:54.465011 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:54.465023 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:12:54.465151 (+ 128us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:12:54.465152 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:12:54.465181 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 23:12:54.465242 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260620 23:12:54.467199 18726 raft_consensus.cc:1261] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Refusing update from remote peer a1316ced4e934746b29fba8e79522b9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:54.467207 18866 raft_consensus.cc:1261] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Refusing update from remote peer a1316ced4e934746b29fba8e79522b9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:54.467442 19032 consensus_queue.cc:1035] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eb8ca9ef5de4435bde4bec9d837acd2" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:54.467571 19045 consensus_queue.cc:1035] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a1316ced4e934746b29fba8e79522b9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80a26c5c8c745249b0786fb3f708b69" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 40837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:54.474514 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:54.472360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:48649 (local address 127.16.79.1:34503)
0620 23:12:54.472395 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:54.472396 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:54.472406 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:12:54.472431 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:54.472431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:54.472439 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:54.472481 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:54.472552 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.473055 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.473942 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.474210 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.474274 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:54.474377 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:54.474398 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:54.474431 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:54.474461 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 23:12:54.474473 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:12:54.474536 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:54.472186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:34503 (local address 127.16.79.2:48649)
0620 23:12:54.472316 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:54.472326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:54.472371 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:54.472486 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:54.472486 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:54.472487 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:12:54.472517 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:54.472518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.473118 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.473119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:54.473831 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:54.473834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:54.474281 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:54.474282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:54.474320 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:54.474324 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:12:54.474430 (+ 106us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:12:54.474430 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 23:12:54.474460 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 23:12:54.474487 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":101,"threads_started":1}
W20260620 23:12:54.474888 18997 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:12:55.994594 18429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:55.994722 18429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:55.995554 18429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b653a0aa1cd41619320f280704b6594: Sending DeleteTablet for 3 replicas of tablet 2ff9ffe5e2b949018aefbc1fb8d776e2
I20260620 23:12:55.995806 18846 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff9ffe5e2b949018aefbc1fb8d776e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:55 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:44060
I20260620 23:12:55.995934 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 18501
I20260620 23:12:55.995996 18705 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff9ffe5e2b949018aefbc1fb8d776e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:55 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:40976
I20260620 23:12:55.996017 19077 tablet_replica.cc:330] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69: stopping tablet replica
I20260620 23:12:55.996624 19077 raft_consensus.cc:2229] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:55.996680 19077 raft_consensus.cc:2258] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:55.996984 19077 ts_tablet_manager.cc:1905] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:55.997732 19078 tablet_replica.cc:330] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2: stopping tablet replica
I20260620 23:12:55.997814 19078 raft_consensus.cc:2229] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:55.997884 19078 raft_consensus.cc:2258] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:55.998143 19078 ts_tablet_manager.cc:1905] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:55.998962 19077 ts_tablet_manager.cc:1918] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:12:55.999040 19077 log.cc:1196] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/2ff9ffe5e2b949018aefbc1fb8d776e2
I20260620 23:12:55.999262 19077 ts_tablet_manager.cc:1939] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P a80a26c5c8c745249b0786fb3f708b69: Deleting consensus metadata
I20260620 23:12:55.999505 19078 ts_tablet_manager.cc:1918] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:12:55.999583 19078 log.cc:1196] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/2ff9ffe5e2b949018aefbc1fb8d776e2
I20260620 23:12:55.999622 18416 catalog_manager.cc:4841] TS a80a26c5c8c745249b0786fb3f708b69 (127.16.79.3:40837): tablet 2ff9ffe5e2b949018aefbc1fb8d776e2 (table test-table [id=7c2959ed70794425baa2c9d185504fb4]) successfully deleted
I20260620 23:12:55.999850 19078 ts_tablet_manager.cc:1939] T 2ff9ffe5e2b949018aefbc1fb8d776e2 P 0eb8ca9ef5de4435bde4bec9d837acd2: Deleting consensus metadata
I20260620 23:12:56.000188 18415 catalog_manager.cc:4841] TS 0eb8ca9ef5de4435bde4bec9d837acd2 (127.16.79.2:34951): tablet 2ff9ffe5e2b949018aefbc1fb8d776e2 (table test-table [id=7c2959ed70794425baa2c9d185504fb4]) successfully deleted
W20260620 23:12:56.002868 18415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:34503 (error 108)
I20260620 23:12:56.003168 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 18641
I20260620 23:12:56.005000 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:56.003238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:34503 (local address 127.0.0.1:57774)
0620 23:12:56.004926 (+ 1688us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:56.004969 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:34503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1667,"thread_start_us":75,"threads_started":1}
W20260620 23:12:56.005141 18415 catalog_manager.cc:4584] TS a1316ced4e934746b29fba8e79522b9f (127.16.79.1:34503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ff9ffe5e2b949018aefbc1fb8d776e2: Network error: Client connection negotiation failed: client connection to 127.16.79.1:34503: connect: Connection refused (error 111)
I20260620 23:12:56.008989 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 18781
I20260620 23:12:56.014458 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 18397
2026-06-20T23:12:56Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5822 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:12:56 dist-test-slave-pswb krb5kdc[19083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:12:56 dist-test-slave-pswb krb5kdc[19083](info): set up 2 sockets
Jun 20 23:12:56 dist-test-slave-pswb krb5kdc[19083](info): commencing operation
krb5kdc: starting...
W20260620 23:12:57.060660 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:12:57 dist-test-slave-pswb krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:12:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:12:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:12:57.183524 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:42301
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:42301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:57.266103 19096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:57.266301 19096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:57.266324 19096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:57.267897 19096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:12:57.267937 19096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:57.267951 19096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:57.267964 19096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:12:57.267976 19096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:12:57.269738 19096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:42301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:42301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:57.269989 19096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:57.270231 19096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:57.275499 19102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:57.275503 19104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:57.275520 19096 server_base.cc:1034] running on GCE node
W20260620 23:12:57.275503 19101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:57.275874 19096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:57.276104 19096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:57.277292 19096 hybrid_clock.cc:648] HybridClock initialized: now 1781997177277279 us; error 47 us; skew 500 ppm
Jun 20 23:12:57 dist-test-slave-pswb krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997177, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:57.279917 19096 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:12:57.280390 19096 webserver.cc:460] Webserver started at http://127.16.79.62:39153/ using document root <none> and password file <none>
I20260620 23:12:57.280627 19096 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:57.280695 19096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:57.280872 19096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:57.281903 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "3ca2f683ba324dfba3ab2ce7d3f20ded"
format_stamp: "Formatted at 2026-06-20 23:12:57 on dist-test-slave-pswb"
server_key: "5147004b894a8d2cc025a7ecb058ff5a"
server_key_iv: "29a3d6f2cecea371a1b680541f281e71"
server_key_version: "encryptionkey@0"
I20260620 23:12:57.282254 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "3ca2f683ba324dfba3ab2ce7d3f20ded"
format_stamp: "Formatted at 2026-06-20 23:12:57 on dist-test-slave-pswb"
server_key: "5147004b894a8d2cc025a7ecb058ff5a"
server_key_iv: "29a3d6f2cecea371a1b680541f281e71"
server_key_version: "encryptionkey@0"
I20260620 23:12:57.283660 19096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:12:57.284938 19111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:57.285235 19096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:12:57.285430 19096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "3ca2f683ba324dfba3ab2ce7d3f20ded"
format_stamp: "Formatted at 2026-06-20 23:12:57 on dist-test-slave-pswb"
server_key: "5147004b894a8d2cc025a7ecb058ff5a"
server_key_iv: "29a3d6f2cecea371a1b680541f281e71"
server_key_version: "encryptionkey@0"
I20260620 23:12:57.285548 19096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:57.311556 19096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:57.313874 19096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:57.314035 19096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:57.318549 19096 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:42301
I20260620 23:12:57.318593 19163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:42301 every 8 connection(s)
I20260620 23:12:57.318925 19096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:12:57.319816 19164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:57.322263 19164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded: Bootstrap starting.
I20260620 23:12:57.322872 19164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:57.323184 19164 log.cc:824] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:57.323933 19164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded: No bootstrap required, opened a new log
I20260620 23:12:57.325618 19164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca2f683ba324dfba3ab2ce7d3f20ded" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 42301 } }
I20260620 23:12:57.325763 19164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:57.325790 19164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca2f683ba324dfba3ab2ce7d3f20ded, State: Initialized, Role: FOLLOWER
I20260620 23:12:57.325932 19164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca2f683ba324dfba3ab2ce7d3f20ded" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 42301 } }
I20260620 23:12:57.326001 19164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:12:57.326022 19164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:12:57.326082 19164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:57.326573 19164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca2f683ba324dfba3ab2ce7d3f20ded" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 42301 } }
I20260620 23:12:57.326663 19164 leader_election.cc:302] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ca2f683ba324dfba3ab2ce7d3f20ded; no voters:
I20260620 23:12:57.326862 19164 leader_election.cc:288] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:12:57.326911 19167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:57.327121 19164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:12:57.327287 19167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [term 1 LEADER]: Becoming Leader. State: Replica: 3ca2f683ba324dfba3ab2ce7d3f20ded, State: Running, Role: LEADER
I20260620 23:12:57.327400 19167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca2f683ba324dfba3ab2ce7d3f20ded" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 42301 } }
I20260620 23:12:57.327791 19169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ca2f683ba324dfba3ab2ce7d3f20ded. Latest consensus state: current_term: 1 leader_uuid: "3ca2f683ba324dfba3ab2ce7d3f20ded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca2f683ba324dfba3ab2ce7d3f20ded" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 42301 } } }
I20260620 23:12:57.327884 19169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:57.328213 19168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ca2f683ba324dfba3ab2ce7d3f20ded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca2f683ba324dfba3ab2ce7d3f20ded" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 42301 } } }
I20260620 23:12:57.328253 19176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:12:57.328292 19168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:57.328755 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 19096
I20260620 23:12:57.328841 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:12:57.329016 16700 external_mini_cluster.cc:1428] Setting key 7b6d2a61a360a706ea0f8dc69a72d570
I20260620 23:12:57.329190 19176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:12:57.333128 19176 catalog_manager.cc:1347] Generated new cluster ID: 53ffafcc9941467a9017398a1821e728
I20260620 23:12:57.333194 19176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:12:57 dist-test-slave-pswb krb5kdc[19083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:57.339797 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:57.329663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.16.79.62:42301)
0620 23:12:57.329932 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:57.329945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:57.330115 (+ 170us) server_negotiation.cc:407] Connection header received
0620 23:12:57.330337 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:57.330347 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:57.330417 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:57.330598 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:57.331317 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.332369 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.333354 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.333836 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.334069 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:57.336544 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:57.336547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:57.336548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:57.336558 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:57.338215 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:57.338634 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:57.338637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:57.338638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:57.338727 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:57.338942 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:57.338943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:57.338944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:57.339181 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:57.339353 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:57.339447 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 23:12:57.339593 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":140,"threads_started":1}
I20260620 23:12:57.361429 19176 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:12:57.361979 19176 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:12:57.367039 19176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ca2f683ba324dfba3ab2ce7d3f20ded: Generated new TSK 0
I20260620 23:12:57.367213 19176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:12:57.397735 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:42301
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:57.499996 19190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:57.500188 19190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:57.500211 19190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:57.501353 19190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:57.501392 19190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:57.501418 19190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:12:57.502740 19190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:42301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:57.502910 19190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:57.503075 19190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:57.508255 19198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:57.508311 19195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:57.508311 19196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:57.508334 19190 server_base.cc:1034] running on GCE node
I20260620 23:12:57.508590 19190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:57.508837 19190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:57.509980 19190 hybrid_clock.cc:648] HybridClock initialized: now 1781997177509967 us; error 29 us; skew 500 ppm
Jun 20 23:12:57 dist-test-slave-pswb krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997177, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:57.512910 19190 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:12:57.513336 19190 webserver.cc:460] Webserver started at http://127.16.79.1:38029/ using document root <none> and password file <none>
I20260620 23:12:57.513585 19190 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:57.513655 19190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:57.513800 19190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:57.514829 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "780e457d53b841b8be241a341e28bee5"
format_stamp: "Formatted at 2026-06-20 23:12:57 on dist-test-slave-pswb"
server_key: "48a5d9cfe95b20f71af2ad9926d5b22a"
server_key_iv: "1bbe44f748ae072d7b2e96d067f16c17"
server_key_version: "encryptionkey@0"
I20260620 23:12:57.515174 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "780e457d53b841b8be241a341e28bee5"
format_stamp: "Formatted at 2026-06-20 23:12:57 on dist-test-slave-pswb"
server_key: "48a5d9cfe95b20f71af2ad9926d5b22a"
server_key_iv: "1bbe44f748ae072d7b2e96d067f16c17"
server_key_version: "encryptionkey@0"
I20260620 23:12:57.516757 19190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:12:57.517540 19205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:57.517709 19190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:12:57.517799 19190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "780e457d53b841b8be241a341e28bee5"
format_stamp: "Formatted at 2026-06-20 23:12:57 on dist-test-slave-pswb"
server_key: "48a5d9cfe95b20f71af2ad9926d5b22a"
server_key_iv: "1bbe44f748ae072d7b2e96d067f16c17"
server_key_version: "encryptionkey@0"
I20260620 23:12:57.517879 19190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:57.537977 19190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:57.540205 19190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:57.540381 19190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:57.540616 19190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:57.540925 19190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:57.540978 19190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:57.541019 19190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:57.541046 19190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:57.547616 19190 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:33661
I20260620 23:12:57.547694 19318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:33661 every 8 connection(s)
I20260620 23:12:57.547998 19190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:12:57 dist-test-slave-pswb krb5kdc[19083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997177, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:57.556883 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 19190
I20260620 23:12:57.556981 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:12:57.557145 16700 external_mini_cluster.cc:1428] Setting key 628ff3e5c3710add30d887b30cff9800
I20260620 23:12:57.557569 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:57.548835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:56841 (local address 127.16.79.62:42301)
0620 23:12:57.548885 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:57.548887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:57.549004 (+ 117us) server_negotiation.cc:407] Connection header received
0620 23:12:57.549461 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:57.549461 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:57.549473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:57.549525 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:57.550414 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.550908 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.551956 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.552255 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.552364 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:57.554654 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:57.554655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:57.554656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:57.554662 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:57.555874 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:57.556247 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:57.556247 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:57.556248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:57.556299 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:57.556509 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:57.556509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:57.556510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:57.556616 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:57.556671 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:57.557280 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 23:12:57.557445 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:12:57.557791 19321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:57.548704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:42301 (local address 127.16.79.1:56841)
0620 23:12:57.548896 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:57.548931 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:57.549377 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:57.549595 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:57.549598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:57.550141 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:57.550324 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:57.550328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.550990 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.550990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:57.551786 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:57.551788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.552376 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.552377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:57.552476 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:57.553118 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:57.553126 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:57.554549 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:57.555956 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:57.555962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:57.555963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:57.556175 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:57.556362 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:57.556362 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:57.556362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:57.556450 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:57.556816 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:57.556827 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:57.557059 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 23:12:57.557224 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 23:12:57.557492 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260620 23:12:57.558492 19319 heartbeater.cc:347] Connected to a master server at 127.16.79.62:42301
I20260620 23:12:57.558616 19319 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:57.558804 19319 heartbeater.cc:511] Master 127.16.79.62:42301 requested a full tablet report, sending...
I20260620 23:12:57.559362 19128 ts_manager.cc:194] Registered new tserver with Master: 780e457d53b841b8be241a341e28bee5 (127.16.79.1:33661)
I20260620 23:12:57.560000 19128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:56841
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:12:57.585562 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:42301
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:57.669180 19324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:57.669381 19324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:57.669405 19324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:57.670619 19324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:57.670662 19324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:57.670688 19324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:12:57.672128 19324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:42301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:57.672349 19324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:57.672580 19324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:57.677263 19330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:57.677291 19332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:57.677282 19329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:57.677505 19324 server_base.cc:1034] running on GCE node
I20260620 23:12:57.677709 19324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:57.677897 19324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:57.679029 19324 hybrid_clock.cc:648] HybridClock initialized: now 1781997177679018 us; error 27 us; skew 500 ppm
Jun 20 23:12:57 dist-test-slave-pswb krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997177, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:57.681460 19324 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:12:57.681861 19324 webserver.cc:460] Webserver started at http://127.16.79.2:36495/ using document root <none> and password file <none>
I20260620 23:12:57.682075 19324 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:57.682117 19324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:57.682276 19324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:57.683257 19324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "7ec399d5e3ef4195b1d7432c5e5fada4"
format_stamp: "Formatted at 2026-06-20 23:12:57 on dist-test-slave-pswb"
server_key: "e0d7a794a209f4783974b5c7d5aa5898"
server_key_iv: "ebdf559d297a23f6051547b03247a41a"
server_key_version: "encryptionkey@0"
I20260620 23:12:57.683606 19324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "7ec399d5e3ef4195b1d7432c5e5fada4"
format_stamp: "Formatted at 2026-06-20 23:12:57 on dist-test-slave-pswb"
server_key: "e0d7a794a209f4783974b5c7d5aa5898"
server_key_iv: "ebdf559d297a23f6051547b03247a41a"
server_key_version: "encryptionkey@0"
I20260620 23:12:57.685065 19324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:12:57.686080 19339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:57.686273 19324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:12:57.686381 19324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "7ec399d5e3ef4195b1d7432c5e5fada4"
format_stamp: "Formatted at 2026-06-20 23:12:57 on dist-test-slave-pswb"
server_key: "e0d7a794a209f4783974b5c7d5aa5898"
server_key_iv: "ebdf559d297a23f6051547b03247a41a"
server_key_version: "encryptionkey@0"
I20260620 23:12:57.686460 19324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:57.696484 19324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:57.698468 19324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:57.698647 19324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:57.698864 19324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:57.699175 19324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:57.699234 19324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:57.699275 19324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:57.699303 19324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:57.704891 19324 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:43383
I20260620 23:12:57.705004 19452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:43383 every 8 connection(s)
I20260620 23:12:57.705255 19324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:12:57.710530 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 19324
I20260620 23:12:57.710635 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:12:57.710820 16700 external_mini_cluster.cc:1428] Setting key cafd8dbe8823de52135e9fedff8072b2
Jun 20 23:12:57 dist-test-slave-pswb krb5kdc[19083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997177, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:57.715240 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:57.706053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:37297 (local address 127.16.79.62:42301)
0620 23:12:57.706094 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:57.706095 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:57.706326 (+ 231us) server_negotiation.cc:407] Connection header received
0620 23:12:57.706761 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:57.706762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:57.706772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:57.706832 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:57.707686 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.708174 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.709193 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.709381 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.709447 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:57.711589 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:57.711590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:57.711591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:57.711598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:57.713207 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:57.713965 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:57.713966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:57.713966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:57.714021 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:57.714355 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:57.714356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:57.714356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:57.714453 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:57.714512 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:57.715048 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 23:12:57.715134 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 23:12:57.715461 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:57.706026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:42301 (local address 127.16.79.2:37297)
0620 23:12:57.706180 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:57.706220 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:57.706680 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:57.706880 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:57.706883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:57.707413 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:57.707589 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:57.707593 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.708274 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.708275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:57.709122 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:57.709124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.709461 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.709462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:57.709555 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:57.710089 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:57.710099 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:57.711501 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:57.713351 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:57.713364 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:57.713368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:57.713846 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:57.714111 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:57.714112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:57.714113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:57.714276 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:57.714550 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:57.714557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:57.714755 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 23:12:57.715025 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 23:12:57.715213 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260620 23:12:57.715960 19453 heartbeater.cc:347] Connected to a master server at 127.16.79.62:42301
I20260620 23:12:57.716053 19453 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:57.716228 19453 heartbeater.cc:511] Master 127.16.79.62:42301 requested a full tablet report, sending...
I20260620 23:12:57.716635 19128 ts_manager.cc:194] Registered new tserver with Master: 7ec399d5e3ef4195b1d7432c5e5fada4 (127.16.79.2:43383)
I20260620 23:12:57.717202 19128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:37297
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:12:57.739203 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:42301
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:57.820372 19458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:57.820576 19458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:57.820605 19458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:57.821936 19458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:57.821980 19458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:57.822007 19458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:12:57.823459 19458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:42301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:57.823645 19458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:57.823879 19458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:57.829020 19464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:57.829173 19463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:57.829015 19466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:57.829263 19458 server_base.cc:1034] running on GCE node
I20260620 23:12:57.829439 19458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:57.829670 19458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:57.830830 19458 hybrid_clock.cc:648] HybridClock initialized: now 1781997177830812 us; error 30 us; skew 500 ppm
Jun 20 23:12:57 dist-test-slave-pswb krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997177, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:57.833320 19458 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:12:57.833734 19458 webserver.cc:460] Webserver started at http://127.16.79.3:36253/ using document root <none> and password file <none>
I20260620 23:12:57.833956 19458 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:57.834022 19458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:57.834153 19458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:57.835469 19458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "c72dcc5de82049f98ec69da9b25ba203"
format_stamp: "Formatted at 2026-06-20 23:12:57 on dist-test-slave-pswb"
server_key: "a600fa8e2a4b461bb41244254ef43db3"
server_key_iv: "a1323e6d7a220f36ed277a297d207326"
server_key_version: "encryptionkey@0"
I20260620 23:12:57.835850 19458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "c72dcc5de82049f98ec69da9b25ba203"
format_stamp: "Formatted at 2026-06-20 23:12:57 on dist-test-slave-pswb"
server_key: "a600fa8e2a4b461bb41244254ef43db3"
server_key_iv: "a1323e6d7a220f36ed277a297d207326"
server_key_version: "encryptionkey@0"
I20260620 23:12:57.837275 19458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:12:57.838047 19473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:57.838217 19458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:12:57.838317 19458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "c72dcc5de82049f98ec69da9b25ba203"
format_stamp: "Formatted at 2026-06-20 23:12:57 on dist-test-slave-pswb"
server_key: "a600fa8e2a4b461bb41244254ef43db3"
server_key_iv: "a1323e6d7a220f36ed277a297d207326"
server_key_version: "encryptionkey@0"
I20260620 23:12:57.838397 19458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:57.846102 19458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:57.848179 19458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:57.848429 19458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:57.848949 19458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:57.849632 19458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:57.849710 19458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:57.849752 19458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:57.849784 19458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:57.855976 19458 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:33345
I20260620 23:12:57.856015 19586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:33345 every 8 connection(s)
I20260620 23:12:57.856323 19458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:12:57 dist-test-slave-pswb krb5kdc[19083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997177, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:57.863777 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 19458
I20260620 23:12:57.863889 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:12:57.864095 16700 external_mini_cluster.cc:1428] Setting key 8c2ad0a400616c319e386e0f64de1799
I20260620 23:12:57.866146 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:57.857001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:53173 (local address 127.16.79.62:42301)
0620 23:12:57.857084 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:57.857087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:57.857259 (+ 172us) server_negotiation.cc:407] Connection header received
0620 23:12:57.857778 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:57.857780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:57.857792 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:57.857838 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:57.859223 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.859730 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.860798 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.861005 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.861070 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:57.863382 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:57.863383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:57.863383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:57.863389 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:57.865113 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:57.865467 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:57.865468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:57.865468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:57.865517 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:57.865683 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:57.865684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:57.865684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:57.865786 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:57.865834 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:57.866030 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 23:12:57.866085 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:12:57.866320 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:57.856961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:42301 (local address 127.16.79.3:53173)
0620 23:12:57.857134 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:57.857180 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:57.857663 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:57.857907 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:57.857912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:57.858765 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:57.859096 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:57.859103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.859845 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.859846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:57.860668 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:57.860670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:57.861150 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:57.861151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:57.861299 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:57.861845 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:57.861853 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:57.863290 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:57.865185 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:57.865190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:57.865192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:57.865407 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:57.865560 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:57.865561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:57.865561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:57.865639 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:57.865825 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:57.865828 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:57.865931 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 23:12:57.866005 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 23:12:57.866118 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260620 23:12:57.866741 19587 heartbeater.cc:347] Connected to a master server at 127.16.79.62:42301
I20260620 23:12:57.866815 19587 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:57.867022 19587 heartbeater.cc:511] Master 127.16.79.62:42301 requested a full tablet report, sending...
I20260620 23:12:57.867385 19128 ts_manager.cc:194] Registered new tserver with Master: c72dcc5de82049f98ec69da9b25ba203 (127.16.79.3:33345)
I20260620 23:12:57.867782 19128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:53173
I20260620 23:12:57.876353 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:12:57.887480 19186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:57.886529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.16.79.62:42301)
0620 23:12:57.886627 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:57.886629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:57.886713 (+ 84us) server_negotiation.cc:407] Connection header received
0620 23:12:57.886806 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:57.886807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:57.886817 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:57.886891 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:57.887401 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46440: BlockingRecv error: recv got EOF from 127.0.0.1:46440 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:12:57.888453 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 19190
I20260620 23:12:57.893088 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 19324
I20260620 23:12:57.898093 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 19458
I20260620 23:12:57.902100 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 19096
2026-06-20T23:12:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1886 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:12:57 dist-test-slave-pswb krb5kdc[19599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:12:57 dist-test-slave-pswb krb5kdc[19599](info): set up 2 sockets
Jun 20 23:12:57 dist-test-slave-pswb krb5kdc[19599](info): commencing operation
krb5kdc: starting...
W20260620 23:12:58.941133 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:12:59 dist-test-slave-pswb krb5kdc[19599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:12:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:12:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:12:59.060395 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36775
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:36775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:59.141866 19612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:59.142082 19612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:59.142118 19612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:59.143256 19612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:12:59.143322 19612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:59.143349 19612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:59.143378 19612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:12:59.143405 19612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:12:59.145548 19612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:36775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:12:59.145947 19612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:59.146281 19612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:59.151624 19618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:59.151638 19620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:59.151715 19612 server_base.cc:1034] running on GCE node
W20260620 23:12:59.151629 19617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:59.152088 19612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:59.152379 19612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:59.153538 19612 hybrid_clock.cc:648] HybridClock initialized: now 1781997179153507 us; error 44 us; skew 500 ppm
Jun 20 23:12:59 dist-test-slave-pswb krb5kdc[19599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997179, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:59.155850 19612 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:12:59.156306 19612 webserver.cc:460] Webserver started at http://127.16.79.62:36783/ using document root <none> and password file <none>
I20260620 23:12:59.156538 19612 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:59.156603 19612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:59.156774 19612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:59.157883 19612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "31082b6402604f3bb861ec1bc0ecf9d5"
format_stamp: "Formatted at 2026-06-20 23:12:59 on dist-test-slave-pswb"
server_key: "57cdf131debc595fb44d6c3188ad14a1"
server_key_iv: "84f4acbb6182bd91eb5a6519ecfc9682"
server_key_version: "encryptionkey@0"
I20260620 23:12:59.158247 19612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "31082b6402604f3bb861ec1bc0ecf9d5"
format_stamp: "Formatted at 2026-06-20 23:12:59 on dist-test-slave-pswb"
server_key: "57cdf131debc595fb44d6c3188ad14a1"
server_key_iv: "84f4acbb6182bd91eb5a6519ecfc9682"
server_key_version: "encryptionkey@0"
I20260620 23:12:59.159516 19612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:12:59.160214 19627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:59.160382 19612 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:12:59.160476 19612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "31082b6402604f3bb861ec1bc0ecf9d5"
format_stamp: "Formatted at 2026-06-20 23:12:59 on dist-test-slave-pswb"
server_key: "57cdf131debc595fb44d6c3188ad14a1"
server_key_iv: "84f4acbb6182bd91eb5a6519ecfc9682"
server_key_version: "encryptionkey@0"
I20260620 23:12:59.160555 19612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:59.180900 19612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:59.182567 19612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:59.182718 19612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:59.186619 19612 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:36775
I20260620 23:12:59.186650 19679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:36775 every 8 connection(s)
I20260620 23:12:59.187104 19612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:12:59.187680 19680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:59.189790 19680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5: Bootstrap starting.
I20260620 23:12:59.190308 19680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:59.190538 19680 log.cc:824] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:59.191105 19680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5: No bootstrap required, opened a new log
I20260620 23:12:59.192581 19680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31082b6402604f3bb861ec1bc0ecf9d5" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36775 } }
I20260620 23:12:59.192724 19680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:59.192747 19680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31082b6402604f3bb861ec1bc0ecf9d5, State: Initialized, Role: FOLLOWER
I20260620 23:12:59.192838 19680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31082b6402604f3bb861ec1bc0ecf9d5" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36775 } }
I20260620 23:12:59.192919 19680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:12:59.192947 19680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:12:59.192991 19680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:59.193806 19680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31082b6402604f3bb861ec1bc0ecf9d5" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36775 } }
I20260620 23:12:59.193886 19680 leader_election.cc:302] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31082b6402604f3bb861ec1bc0ecf9d5; no voters:
I20260620 23:12:59.194074 19680 leader_election.cc:288] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:12:59.194106 19683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:59.194324 19680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:12:59.194322 19683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [term 1 LEADER]: Becoming Leader. State: Replica: 31082b6402604f3bb861ec1bc0ecf9d5, State: Running, Role: LEADER
I20260620 23:12:59.194446 19683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31082b6402604f3bb861ec1bc0ecf9d5" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36775 } }
I20260620 23:12:59.194832 19685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31082b6402604f3bb861ec1bc0ecf9d5. Latest consensus state: current_term: 1 leader_uuid: "31082b6402604f3bb861ec1bc0ecf9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31082b6402604f3bb861ec1bc0ecf9d5" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36775 } } }
I20260620 23:12:59.194973 19685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:59.195215 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 19612
I20260620 23:12:59.195315 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:12:59.195525 16700 external_mini_cluster.cc:1428] Setting key 7de7db1bf49673759e67461ba2873e8b
I20260620 23:12:59.194857 19684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31082b6402604f3bb861ec1bc0ecf9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31082b6402604f3bb861ec1bc0ecf9d5" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36775 } } }
I20260620 23:12:59.195885 19684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5 [sys.catalog]: This master's current role is: LEADER
W20260620 23:12:59.196079 19698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:12:59.196136 19698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:12:59.196262 19699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:12:59.196447 19699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:12:59.197971 19699 catalog_manager.cc:1347] Generated new cluster ID: c790f809a37148429fa01419ac0717bb
I20260620 23:12:59.198035 19699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:12:59 dist-test-slave-pswb krb5kdc[19599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:12:59.209425 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:59.196077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.16.79.62:36775)
0620 23:12:59.196244 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:59.196252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:59.196297 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:12:59.196408 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:59.196415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:59.196468 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:59.196620 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:59.197374 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:59.198711 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:59.199728 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:59.200344 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:59.200453 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:59.206013 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:59.206016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:59.206017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:59.206028 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:59.207830 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:59.208212 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:59.208215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:59.208215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:59.208315 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:59.208524 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:59.208525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:59.208526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:59.208818 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:59.208955 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:59.209058 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 23:12:59.209199 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260620 23:12:59.209921 19699 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:12:59.210338 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:12:59.210577 19699 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:12:59.215394 19699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31082b6402604f3bb861ec1bc0ecf9d5: Generated new TSK 0
I20260620 23:12:59.215538 19699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:12:59.226496 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:59.223122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.16.79.62:36775)
0620 23:12:59.223182 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:59.223184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:59.223301 (+ 117us) server_negotiation.cc:407] Connection header received
0620 23:12:59.223376 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:59.223377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:59.223388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:59.223443 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:59.223765 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:59.224271 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:59.225116 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:59.225334 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:59.225406 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:59.226036 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:59.226037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:59.226037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:59.226214 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:59.226228 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:59.226233 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:59.226301 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:59.226387 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 23:12:59.226408 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:12:59.228542 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 19612
2026-06-20T23:12:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1323 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:12:59 dist-test-slave-pswb krb5kdc[19713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:12:59 dist-test-slave-pswb krb5kdc[19713](info): set up 2 sockets
Jun 20 23:12:59 dist-test-slave-pswb krb5kdc[19713](info): commencing operation
krb5kdc: starting...
W20260620 23:13:00.284619 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:13:00 dist-test-slave-pswb krb5kdc[19713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:13:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:13:00.408551 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:35847
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:35847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:00.485356 19726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:00.485584 19726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:00.485620 19726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:00.486720 19726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:00.486788 19726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:00.486812 19726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:00.486840 19726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:00.486871 19726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:00.488274 19726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:35847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:35847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:00.488534 19726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:00.488777 19726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:00.493507 19732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:00.493517 19731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:00.493554 19726 server_base.cc:1034] running on GCE node
W20260620 23:13:00.493552 19734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:00.494016 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:00.494293 19726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:00.495461 19726 hybrid_clock.cc:648] HybridClock initialized: now 1781997180495447 us; error 48 us; skew 500 ppm
Jun 20 23:13:00 dist-test-slave-pswb krb5kdc[19713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997180, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:00.498108 19726 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:13:00.498627 19726 webserver.cc:460] Webserver started at http://127.16.79.62:42871/ using document root <none> and password file <none>
I20260620 23:13:00.498847 19726 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:00.498914 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:00.499060 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:00.500037 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "b3fd6fe43f6f45319cb1dec0b1c8a401"
format_stamp: "Formatted at 2026-06-20 23:13:00 on dist-test-slave-pswb"
server_key: "17256a254848087aa852188c0aa8fe0c"
server_key_iv: "57485609aa9c2aac4910aa9cb23afb98"
server_key_version: "encryptionkey@0"
I20260620 23:13:00.500393 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "b3fd6fe43f6f45319cb1dec0b1c8a401"
format_stamp: "Formatted at 2026-06-20 23:13:00 on dist-test-slave-pswb"
server_key: "17256a254848087aa852188c0aa8fe0c"
server_key_iv: "57485609aa9c2aac4910aa9cb23afb98"
server_key_version: "encryptionkey@0"
I20260620 23:13:00.501770 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:13:00.502530 19741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:00.502807 19726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:00.502925 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "b3fd6fe43f6f45319cb1dec0b1c8a401"
format_stamp: "Formatted at 2026-06-20 23:13:00 on dist-test-slave-pswb"
server_key: "17256a254848087aa852188c0aa8fe0c"
server_key_iv: "57485609aa9c2aac4910aa9cb23afb98"
server_key_version: "encryptionkey@0"
I20260620 23:13:00.503006 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:00.521896 19726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:00.523835 19726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:00.524000 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:00.528304 19726 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:35847
I20260620 23:13:00.528528 19793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:35847 every 8 connection(s)
I20260620 23:13:00.528764 19726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:00.529896 19794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:00.532258 19794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401: Bootstrap starting.
I20260620 23:13:00.532785 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 19726
I20260620 23:13:00.532899 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:00.533062 19794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:00.533109 16700 external_mini_cluster.cc:1428] Setting key 3d0f400f62622250827832a62082d426
I20260620 23:13:00.533399 19794 log.cc:824] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:00.534219 19794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401: No bootstrap required, opened a new log
I20260620 23:13:00.536841 19794 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fd6fe43f6f45319cb1dec0b1c8a401" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35847 } }
I20260620 23:13:00.537007 19794 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:00.537041 19794 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fd6fe43f6f45319cb1dec0b1c8a401, State: Initialized, Role: FOLLOWER
I20260620 23:13:00.537177 19794 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fd6fe43f6f45319cb1dec0b1c8a401" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35847 } }
I20260620 23:13:00.537314 19794 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:00.537364 19794 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:00.537422 19794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:00.538018 19794 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fd6fe43f6f45319cb1dec0b1c8a401" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35847 } }
I20260620 23:13:00.538125 19794 leader_election.cc:302] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3fd6fe43f6f45319cb1dec0b1c8a401; no voters:
I20260620 23:13:00.538276 19794 leader_election.cc:288] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:00.538334 19799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:00.538486 19799 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [term 1 LEADER]: Becoming Leader. State: Replica: b3fd6fe43f6f45319cb1dec0b1c8a401, State: Running, Role: LEADER
I20260620 23:13:00.538525 19794 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:00.538586 19799 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fd6fe43f6f45319cb1dec0b1c8a401" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35847 } }
I20260620 23:13:00.538877 19799 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3fd6fe43f6f45319cb1dec0b1c8a401. Latest consensus state: current_term: 1 leader_uuid: "b3fd6fe43f6f45319cb1dec0b1c8a401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fd6fe43f6f45319cb1dec0b1c8a401" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35847 } } }
I20260620 23:13:00.538964 19799 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:00.538889 19800 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3fd6fe43f6f45319cb1dec0b1c8a401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fd6fe43f6f45319cb1dec0b1c8a401" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35847 } } }
I20260620 23:13:00.539098 19800 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:00.539808 19804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:00.540263 19804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:13:00 dist-test-slave-pswb krb5kdc[19713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:00.542374 19804 catalog_manager.cc:1347] Generated new cluster ID: 56d305aff0ca4ee3ae3567dfb4ea6a20
I20260620 23:13:00.542428 19804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:00.548336 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:00.533605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.16.79.62:35847)
0620 23:13:00.533796 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:00.533806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:00.533839 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:13:00.534481 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:00.534490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:00.534587 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:00.534760 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:00.535356 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:00.537093 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:00.537969 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:00.538643 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:00.539115 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:00.541958 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:00.541961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:00.541962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:00.541971 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:00.546638 (+ 4667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:00.547117 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:00.547120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:00.547121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:00.547224 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:00.547447 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:00.547447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:00.547448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:00.547713 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:00.547859 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:00.548022 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:13:00.548147 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":109,"threads_started":1}
I20260620 23:13:00.569612 19804 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:00.570310 19804 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:00.576290 19804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3fd6fe43f6f45319cb1dec0b1c8a401: Generated new TSK 0
I20260620 23:13:00.576450 19804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:13:00.608922 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:35847
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:00.687929 19820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:00.688117 19820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:00.688139 19820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:00.689287 19820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:00.689327 19820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:00.689352 19820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:00.690814 19820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:35847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:00.691052 19820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:00.691272 19820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:00.696341 19826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:00.696383 19828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:00.696382 19825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:00.696581 19820 server_base.cc:1034] running on GCE node
I20260620 23:13:00.696892 19820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:00.697139 19820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:00.698305 19820 hybrid_clock.cc:648] HybridClock initialized: now 1781997180698291 us; error 28 us; skew 500 ppm
Jun 20 23:13:00 dist-test-slave-pswb krb5kdc[19713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997180, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:00.700925 19820 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:13:00.701335 19820 webserver.cc:460] Webserver started at http://127.16.79.1:41203/ using document root <none> and password file <none>
I20260620 23:13:00.701556 19820 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:00.701623 19820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:00.701766 19820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:00.702773 19820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "51703b6265d8479ca827800985006a10"
format_stamp: "Formatted at 2026-06-20 23:13:00 on dist-test-slave-pswb"
server_key: "cc51ad5ee906b5ca56686ed9fb4e29b5"
server_key_iv: "fc8ecfa7585c10a7254b53d00a808245"
server_key_version: "encryptionkey@0"
I20260620 23:13:00.703126 19820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "51703b6265d8479ca827800985006a10"
format_stamp: "Formatted at 2026-06-20 23:13:00 on dist-test-slave-pswb"
server_key: "cc51ad5ee906b5ca56686ed9fb4e29b5"
server_key_iv: "fc8ecfa7585c10a7254b53d00a808245"
server_key_version: "encryptionkey@0"
I20260620 23:13:00.704607 19820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:13:00.705513 19835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:00.705720 19820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:00.705822 19820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "51703b6265d8479ca827800985006a10"
format_stamp: "Formatted at 2026-06-20 23:13:00 on dist-test-slave-pswb"
server_key: "cc51ad5ee906b5ca56686ed9fb4e29b5"
server_key_iv: "fc8ecfa7585c10a7254b53d00a808245"
server_key_version: "encryptionkey@0"
I20260620 23:13:00.705904 19820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:00.719321 19820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:00.721626 19820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:00.721781 19820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:00.722005 19820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:00.722301 19820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:00.722353 19820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:00.722393 19820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:00.722420 19820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:00.728174 19820 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:40105
I20260620 23:13:00.728210 19948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:40105 every 8 connection(s)
I20260620 23:13:00.728538 19820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:00.734006 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 19820
I20260620 23:13:00.734104 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:13:00.734272 16700 external_mini_cluster.cc:1428] Setting key e67b8774c32c9fe07c4244f3d164039f
Jun 20 23:13:00 dist-test-slave-pswb krb5kdc[19713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997180, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:00.738631 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:00.729555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:48203 (local address 127.16.79.62:35847)
0620 23:13:00.729654 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:00.729656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:00.729785 (+ 129us) server_negotiation.cc:407] Connection header received
0620 23:13:00.730328 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:00.730329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:00.730341 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:00.730389 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:00.731310 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:00.731827 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:00.732867 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:00.733068 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:00.733147 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:00.735580 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:00.735582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:00.735582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:00.735590 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:00.737189 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:00.737662 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:00.737663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:00.737663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:00.737721 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:00.737980 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:00.737981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:00.737981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:00.738078 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:00.738129 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:00.738438 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 23:13:00.738542 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:13:00.738876 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:00.729428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:35847 (local address 127.16.79.1:48203)
0620 23:13:00.729645 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:00.729691 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:00.730222 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:00.730469 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:00.730473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:00.731023 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:00.731212 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:00.731217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:00.731930 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:00.731931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:00.732785 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:00.732787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:00.733163 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:00.733164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:00.733287 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:00.733788 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:00.733795 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:00.735495 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:00.737264 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:00.737269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:00.737271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:00.737583 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:00.737786 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:00.737787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:00.737787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:00.737924 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:00.738138 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:00.738143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:00.738305 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 23:13:00.738418 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:13:00.738607 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260620 23:13:00.739357 19949 heartbeater.cc:347] Connected to a master server at 127.16.79.62:35847
I20260620 23:13:00.739459 19949 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:00.739661 19949 heartbeater.cc:511] Master 127.16.79.62:35847 requested a full tablet report, sending...
I20260620 23:13:00.740125 19758 ts_manager.cc:194] Registered new tserver with Master: 51703b6265d8479ca827800985006a10 (127.16.79.1:40105)
I20260620 23:13:00.740865 19758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:48203
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:13:00.759070 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:35847
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:00.837419 19954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:00.837627 19954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:00.837664 19954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:00.838876 19954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:00.838943 19954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:00.838986 19954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:00.840492 19954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:35847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:00.840766 19954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:00.840977 19954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:00.845974 19959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:00.845949 19962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:00.845993 19954 server_base.cc:1034] running on GCE node
W20260620 23:13:00.845960 19960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:00.846365 19954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:00.846630 19954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:00.847805 19954 hybrid_clock.cc:648] HybridClock initialized: now 1781997180847783 us; error 39 us; skew 500 ppm
Jun 20 23:13:00 dist-test-slave-pswb krb5kdc[19713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997180, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:00.850975 19954 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:13:00.851606 19954 webserver.cc:460] Webserver started at http://127.16.79.2:34503/ using document root <none> and password file <none>
I20260620 23:13:00.851897 19954 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:00.851974 19954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:00.852130 19954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:00.853639 19954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "90842f97bc394c81b244edf65020531c"
format_stamp: "Formatted at 2026-06-20 23:13:00 on dist-test-slave-pswb"
server_key: "53d96a69dbbb2f0d32a15516e8d433b5"
server_key_iv: "bab956d16910c1da946a53b0a56e21e6"
server_key_version: "encryptionkey@0"
I20260620 23:13:00.854228 19954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "90842f97bc394c81b244edf65020531c"
format_stamp: "Formatted at 2026-06-20 23:13:00 on dist-test-slave-pswb"
server_key: "53d96a69dbbb2f0d32a15516e8d433b5"
server_key_iv: "bab956d16910c1da946a53b0a56e21e6"
server_key_version: "encryptionkey@0"
I20260620 23:13:00.855649 19954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:13:00.856468 19969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:00.856669 19954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:00.856801 19954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "90842f97bc394c81b244edf65020531c"
format_stamp: "Formatted at 2026-06-20 23:13:00 on dist-test-slave-pswb"
server_key: "53d96a69dbbb2f0d32a15516e8d433b5"
server_key_iv: "bab956d16910c1da946a53b0a56e21e6"
server_key_version: "encryptionkey@0"
I20260620 23:13:00.856884 19954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:00.868805 19954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:00.870919 19954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:00.871075 19954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:00.871300 19954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:00.871599 19954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:00.871657 19954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:00.871692 19954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:00.871719 19954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:00.877930 19954 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:36407
I20260620 23:13:00.877976 20082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:36407 every 8 connection(s)
I20260620 23:13:00.878270 19954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:13:00 dist-test-slave-pswb krb5kdc[19713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997180, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:00.884150 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 19954
I20260620 23:13:00.884280 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:00.884506 16700 external_mini_cluster.cc:1428] Setting key 79f34043f1910527188b7f3cc2fe199f
I20260620 23:13:00.887421 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:00.878902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:35925 (local address 127.16.79.62:35847)
0620 23:13:00.878943 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:00.878945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:00.879124 (+ 179us) server_negotiation.cc:407] Connection header received
0620 23:13:00.879567 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:00.879567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:00.879576 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:00.879636 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:00.880469 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:00.881017 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:00.881929 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:00.882103 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:00.882152 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:00.884222 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:00.884223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:00.884224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:00.884232 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:00.886028 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:00.886407 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:00.886408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:00.886409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:00.886469 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:00.886780 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:00.886781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:00.886781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:00.886901 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:00.886974 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:00.887244 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:13:00.887333 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:13:00.887588 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:00.878854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:35847 (local address 127.16.79.2:35925)
0620 23:13:00.878998 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:00.879036 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:00.879488 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:00.879686 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:00.879689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:00.880211 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:00.880395 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:00.880399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:00.881089 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:00.881090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:00.881856 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:00.881858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:00.882170 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:00.882171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:00.882271 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:00.882682 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:00.882690 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:00.884126 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:00.886111 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:00.886116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:00.886118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:00.886331 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:00.886551 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:00.886553 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:00.886553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:00.886669 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:00.886980 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:00.886986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:00.887103 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 23:13:00.887206 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 23:13:00.887354 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260620 23:13:00.888015 20083 heartbeater.cc:347] Connected to a master server at 127.16.79.62:35847
I20260620 23:13:00.888110 20083 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:00.888293 20083 heartbeater.cc:511] Master 127.16.79.62:35847 requested a full tablet report, sending...
I20260620 23:13:00.888654 19758 ts_manager.cc:194] Registered new tserver with Master: 90842f97bc394c81b244edf65020531c (127.16.79.2:36407)
I20260620 23:13:00.889216 19758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:35925
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:13:00.911566 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:35847
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:00.991086 20088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:00.991264 20088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:00.991286 20088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:00.992352 20088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:00.992388 20088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:00.992413 20088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:00.993750 20088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:35847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:00.993924 20088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:00.994084 20088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:13:00.998893 20088 server_base.cc:1034] running on GCE node
W20260620 23:13:00.998867 20094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:00.998920 20093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:00.998865 20096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:00.999343 20088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:00.999534 20088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:01.000684 20088 hybrid_clock.cc:648] HybridClock initialized: now 1781997181000669 us; error 29 us; skew 500 ppm
Jun 20 23:13:01 dist-test-slave-pswb krb5kdc[19713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997181, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:01.003299 20088 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:13:01.003706 20088 webserver.cc:460] Webserver started at http://127.16.79.3:43101/ using document root <none> and password file <none>
I20260620 23:13:01.003938 20088 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:01.004006 20088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:01.004146 20088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:01.005203 20088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "41d33062e63943cd96b973e0d141cf64"
format_stamp: "Formatted at 2026-06-20 23:13:01 on dist-test-slave-pswb"
server_key: "6a3d1b9989c96368235beba24277bed0"
server_key_iv: "3169bab590bd3c843efdabd974597fd7"
server_key_version: "encryptionkey@0"
I20260620 23:13:01.005590 20088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "41d33062e63943cd96b973e0d141cf64"
format_stamp: "Formatted at 2026-06-20 23:13:01 on dist-test-slave-pswb"
server_key: "6a3d1b9989c96368235beba24277bed0"
server_key_iv: "3169bab590bd3c843efdabd974597fd7"
server_key_version: "encryptionkey@0"
I20260620 23:13:01.007017 20088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:13:01.007715 20103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:01.007911 20088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:01.008004 20088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "41d33062e63943cd96b973e0d141cf64"
format_stamp: "Formatted at 2026-06-20 23:13:01 on dist-test-slave-pswb"
server_key: "6a3d1b9989c96368235beba24277bed0"
server_key_iv: "3169bab590bd3c843efdabd974597fd7"
server_key_version: "encryptionkey@0"
I20260620 23:13:01.008081 20088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:01.028581 20088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:01.030576 20088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:01.030754 20088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:01.030994 20088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:01.031312 20088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:01.031363 20088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:01.031399 20088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:01.031426 20088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:01.038828 20088 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:38233
I20260620 23:13:01.038909 20216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:38233 every 8 connection(s)
I20260620 23:13:01.039302 20088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:13:01 dist-test-slave-pswb krb5kdc[19713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997181, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:01.047560 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 20088
I20260620 23:13:01.047667 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:01.047873 16700 external_mini_cluster.cc:1428] Setting key 401731b3a3e349420971c188685d94fa
I20260620 23:13:01.048728 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:01.039966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:49429 (local address 127.16.79.62:35847)
0620 23:13:01.040007 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:01.040009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:01.040187 (+ 178us) server_negotiation.cc:407] Connection header received
0620 23:13:01.040800 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:01.040801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:01.040813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:01.040875 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:01.041778 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:01.042275 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:01.043249 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:01.043455 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:01.043529 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:01.045861 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:01.045862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:01.045863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:01.045869 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:01.047151 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:01.047669 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:01.047670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:01.047671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:01.047765 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:01.048067 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:01.048068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:01.048068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:01.048187 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:01.048275 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:01.048538 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 23:13:01.048614 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:13:01.048928 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:01.039936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:35847 (local address 127.16.79.3:49429)
0620 23:13:01.040064 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:01.040110 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:01.040653 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:01.040941 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:01.040945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:01.041498 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:01.041678 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:01.041683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:01.042366 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:01.042367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:01.043152 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:01.043154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:01.043532 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:01.043533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:01.043657 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:01.044299 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:01.044307 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:01.045739 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:01.047257 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:01.047263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:01.047265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:01.047561 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:01.047819 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:01.047819 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:01.047820 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:01.047969 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:01.048260 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:01.048266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:01.048413 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 23:13:01.048518 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:13:01.048664 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
I20260620 23:13:01.049355 20217 heartbeater.cc:347] Connected to a master server at 127.16.79.62:35847
I20260620 23:13:01.049469 20217 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:01.049647 20217 heartbeater.cc:511] Master 127.16.79.62:35847 requested a full tablet report, sending...
I20260620 23:13:01.050083 19758 ts_manager.cc:194] Registered new tserver with Master: 41d33062e63943cd96b973e0d141cf64 (127.16.79.3:38233)
I20260620 23:13:01.050470 19758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:49429
I20260620 23:13:01.060398 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:13:01 dist-test-slave-pswb krb5kdc[19713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997181, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 23:13:01 dist-test-slave-pswb krb5kdc[19713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997181, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:01.084436 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:01.076687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.16.79.62:35847)
0620 23:13:01.076749 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:01.076752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:01.076890 (+ 138us) server_negotiation.cc:407] Connection header received
0620 23:13:01.076986 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:01.076987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:01.076997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:01.077076 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:01.077686 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:01.078216 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:01.079040 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:01.079255 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:01.079349 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:01.081664 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:01.081666 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:01.081666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:01.081674 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:01.083070 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:01.083556 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:01.083557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:01.083557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:01.083617 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:01.083826 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:01.083827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:01.083828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:01.083982 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:01.084097 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:01.084235 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 23:13:01.084323 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 23:13:01.084669 19758 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57614
I20260620 23:13:01.085139 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 19820
I20260620 23:13:01.089800 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 19954
I20260620 23:13:01.094442 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 20088
I20260620 23:13:01.098958 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 19726
2026-06-20T23:13:01Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1873 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:13:01 dist-test-slave-pswb krb5kdc[20229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:13:01 dist-test-slave-pswb krb5kdc[20229](info): set up 2 sockets
Jun 20 23:13:01 dist-test-slave-pswb krb5kdc[20229](info): commencing operation
krb5kdc: starting...
W20260620 23:13:02.153159 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:13:02 dist-test-slave-pswb krb5kdc[20229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:13:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:13:02.278141 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:33895
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:33895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:02.359071 20242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:02.359306 20242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:02.359339 20242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:02.360437 20242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:02.360499 20242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:02.360525 20242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:02.360556 20242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:02.360587 20242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:02.362064 20242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:33895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:33895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:02.362332 20242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:02.362548 20242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:02.367452 20250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:02.367398 20247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:02.367462 20248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:02.367722 20242 server_base.cc:1034] running on GCE node
I20260620 23:13:02.368134 20242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:02.368587 20242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:02.369793 20242 hybrid_clock.cc:648] HybridClock initialized: now 1781997182369782 us; error 64 us; skew 500 ppm
Jun 20 23:13:02 dist-test-slave-pswb krb5kdc[20229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:02.372313 20242 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:13:02.372742 20242 webserver.cc:460] Webserver started at http://127.16.79.62:32883/ using document root <none> and password file <none>
I20260620 23:13:02.372922 20242 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:02.372962 20242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:02.373046 20242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:02.373997 20242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "edebc37cd2554cb6858f67dbe5d625b0"
format_stamp: "Formatted at 2026-06-20 23:13:02 on dist-test-slave-pswb"
server_key: "4eda65b95b4b865b806ea737704b6a6c"
server_key_iv: "ef4d4af52db75a6d62d9764aff1db0cf"
server_key_version: "encryptionkey@0"
I20260620 23:13:02.374287 20242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "edebc37cd2554cb6858f67dbe5d625b0"
format_stamp: "Formatted at 2026-06-20 23:13:02 on dist-test-slave-pswb"
server_key: "4eda65b95b4b865b806ea737704b6a6c"
server_key_iv: "ef4d4af52db75a6d62d9764aff1db0cf"
server_key_version: "encryptionkey@0"
I20260620 23:13:02.375499 20242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:13:02.376295 20257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:02.376479 20242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:02.376564 20242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "edebc37cd2554cb6858f67dbe5d625b0"
format_stamp: "Formatted at 2026-06-20 23:13:02 on dist-test-slave-pswb"
server_key: "4eda65b95b4b865b806ea737704b6a6c"
server_key_iv: "ef4d4af52db75a6d62d9764aff1db0cf"
server_key_version: "encryptionkey@0"
I20260620 23:13:02.376616 20242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:02.392228 20242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:02.394398 20242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:02.394505 20242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:02.398413 20242 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:33895
I20260620 23:13:02.398476 20309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:33895 every 8 connection(s)
I20260620 23:13:02.398912 20242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:02.399909 20310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:02.402899 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 20242
I20260620 23:13:02.402989 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:02.403137 20310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0: Bootstrap starting.
I20260620 23:13:02.403309 16700 external_mini_cluster.cc:1428] Setting key 64f04f937161ac71aa448d1d5a614046
I20260620 23:13:02.403921 20310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:02.404269 20310 log.cc:824] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:02.405058 20310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0: No bootstrap required, opened a new log
I20260620 23:13:02.407334 20310 raft_consensus.cc:348] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edebc37cd2554cb6858f67dbe5d625b0" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33895 } }
I20260620 23:13:02.407508 20310 raft_consensus.cc:374] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:02.407544 20310 raft_consensus.cc:729] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edebc37cd2554cb6858f67dbe5d625b0, State: Initialized, Role: FOLLOWER
I20260620 23:13:02.407709 20310 consensus_queue.cc:260] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edebc37cd2554cb6858f67dbe5d625b0" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33895 } }
I20260620 23:13:02.407799 20310 raft_consensus.cc:388] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:02.407829 20310 raft_consensus.cc:482] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:02.407907 20310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:02.408453 20310 raft_consensus.cc:504] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edebc37cd2554cb6858f67dbe5d625b0" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33895 } }
I20260620 23:13:02.408538 20310 leader_election.cc:302] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edebc37cd2554cb6858f67dbe5d625b0; no voters:
I20260620 23:13:02.408761 20310 leader_election.cc:288] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:02.408887 20315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:02.409047 20310 sys_catalog.cc:564] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:02.409168 20315 raft_consensus.cc:686] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [term 1 LEADER]: Becoming Leader. State: Replica: edebc37cd2554cb6858f67dbe5d625b0, State: Running, Role: LEADER
I20260620 23:13:02.409358 20315 consensus_queue.cc:237] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edebc37cd2554cb6858f67dbe5d625b0" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33895 } }
I20260620 23:13:02.409806 20315 sys_catalog.cc:455] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edebc37cd2554cb6858f67dbe5d625b0. Latest consensus state: current_term: 1 leader_uuid: "edebc37cd2554cb6858f67dbe5d625b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edebc37cd2554cb6858f67dbe5d625b0" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33895 } } }
I20260620 23:13:02.410044 20315 sys_catalog.cc:458] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:02.410739 20316 sys_catalog.cc:455] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edebc37cd2554cb6858f67dbe5d625b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edebc37cd2554cb6858f67dbe5d625b0" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33895 } } }
W20260620 23:13:02.411080 20330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 23:13:02.411278 20316 sys_catalog.cc:458] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0 [sys.catalog]: This master's current role is: LEADER
W20260620 23:13:02.411286 20330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:13:02.411362 20326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:02.411530 20326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:13:02 dist-test-slave-pswb krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:02.413020 20326 catalog_manager.cc:1347] Generated new cluster ID: 72d113a7adeb461b92677dc89f443377
I20260620 23:13:02.413084 20326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:02.416636 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:02.403925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.16.79.62:33895)
0620 23:13:02.404113 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:02.404125 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:02.404159 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:13:02.404771 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:02.404780 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:02.404840 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:02.404994 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:02.405953 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.407369 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.408612 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.409241 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.409736 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:02.412486 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:02.412490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:02.412491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:02.412503 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:02.414968 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:02.415336 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:02.415338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:02.415339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:02.415490 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:02.415754 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:02.415755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:02.415755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:02.415975 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:02.416135 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:02.416261 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 23:13:02.416487 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260620 23:13:02.428717 20326 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:02.429400 20326 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:02.444212 20326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edebc37cd2554cb6858f67dbe5d625b0: Generated new TSK 0
I20260620 23:13:02.444393 20326 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:13:02.473598 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:33895
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:02.558135 20336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:02.558549 20336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:02.558597 20336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:02.560143 20336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:02.560220 20336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:02.560274 20336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:02.562938 20336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:33895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:02.563252 20336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:02.563488 20336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:02.568722 20344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:02.568766 20342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:02.568689 20341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:02.569007 20336 server_base.cc:1034] running on GCE node
I20260620 23:13:02.569204 20336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:02.569440 20336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:02.570643 20336 hybrid_clock.cc:648] HybridClock initialized: now 1781997182570612 us; error 45 us; skew 500 ppm
Jun 20 23:13:02 dist-test-slave-pswb krb5kdc[20229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:02.574110 20336 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:13:02.574537 20336 webserver.cc:460] Webserver started at http://127.16.79.1:42921/ using document root <none> and password file <none>
I20260620 23:13:02.574757 20336 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:02.574826 20336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:02.574980 20336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:02.576012 20336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "429219cc8cd8418c9bc2e60b32c6c8d3"
format_stamp: "Formatted at 2026-06-20 23:13:02 on dist-test-slave-pswb"
server_key: "a40a9e1b9358cd5da62d22576d1751f0"
server_key_iv: "998165473ed29e6de732625650eeb5a9"
server_key_version: "encryptionkey@0"
I20260620 23:13:02.576377 20336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "429219cc8cd8418c9bc2e60b32c6c8d3"
format_stamp: "Formatted at 2026-06-20 23:13:02 on dist-test-slave-pswb"
server_key: "a40a9e1b9358cd5da62d22576d1751f0"
server_key_iv: "998165473ed29e6de732625650eeb5a9"
server_key_version: "encryptionkey@0"
I20260620 23:13:02.577899 20336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:13:02.578778 20351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:02.579195 20336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:02.579381 20336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "429219cc8cd8418c9bc2e60b32c6c8d3"
format_stamp: "Formatted at 2026-06-20 23:13:02 on dist-test-slave-pswb"
server_key: "a40a9e1b9358cd5da62d22576d1751f0"
server_key_iv: "998165473ed29e6de732625650eeb5a9"
server_key_version: "encryptionkey@0"
I20260620 23:13:02.579486 20336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:02.591542 20336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:02.594149 20336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:02.594336 20336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:02.594601 20336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:02.595070 20336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:02.595110 20336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:02.595132 20336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:02.595145 20336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:02.601598 20336 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:34349
I20260620 23:13:02.601640 20464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:34349 every 8 connection(s)
I20260620 23:13:02.601953 20336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:13:02 dist-test-slave-pswb krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:02.609346 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 20336
I20260620 23:13:02.609432 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:13:02.609647 16700 external_mini_cluster.cc:1428] Setting key 8e20b431b972e7778c07087d473d7bda
I20260620 23:13:02.612159 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:02.602717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:57487 (local address 127.16.79.62:33895)
0620 23:13:02.602808 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:02.602810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:02.602960 (+ 150us) server_negotiation.cc:407] Connection header received
0620 23:13:02.603901 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:02.603902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:02.603915 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:02.603972 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:02.604578 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.605165 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.606137 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.606358 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.606444 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:02.609006 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:02.609007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:02.609008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:02.609015 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:02.610810 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:02.611199 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:02.611200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:02.611200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:02.611259 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:02.611489 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:02.611490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:02.611490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:02.611631 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:02.611703 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:02.612031 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 23:13:02.612096 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:13:02.612435 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:02.602666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33895 (local address 127.16.79.1:57487)
0620 23:13:02.602845 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:02.602880 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:02.603784 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:02.604042 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:02.604046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:02.604299 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:02.604482 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:02.604485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.605256 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.605257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:02.606032 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:02.606034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.606454 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.606455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:02.606575 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:02.607141 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:02.607149 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:02.608883 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:02.610897 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:02.610903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:02.610905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:02.611125 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:02.611332 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:02.611333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:02.611333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:02.611425 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:02.611703 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:02.611709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:02.611886 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 23:13:02.612005 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:13:02.612175 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260620 23:13:02.612906 20465 heartbeater.cc:347] Connected to a master server at 127.16.79.62:33895
I20260620 23:13:02.613010 20465 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:02.613180 20465 heartbeater.cc:511] Master 127.16.79.62:33895 requested a full tablet report, sending...
I20260620 23:13:02.614095 20274 ts_manager.cc:194] Registered new tserver with Master: 429219cc8cd8418c9bc2e60b32c6c8d3 (127.16.79.1:34349)
I20260620 23:13:02.614733 20274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:57487
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:13:02.635710 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:33895
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:02.715909 20470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:02.716095 20470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:02.716117 20470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:02.717254 20470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:02.717295 20470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:02.717321 20470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:02.718842 20470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:33895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:02.719069 20470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:02.719300 20470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:02.724196 20478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:02.724247 20476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:02.724211 20475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:02.724414 20470 server_base.cc:1034] running on GCE node
I20260620 23:13:02.724625 20470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:02.724977 20470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:02.726140 20470 hybrid_clock.cc:648] HybridClock initialized: now 1781997182726116 us; error 35 us; skew 500 ppm
Jun 20 23:13:02 dist-test-slave-pswb krb5kdc[20229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:02.729177 20470 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:13:02.730150 20470 webserver.cc:460] Webserver started at http://127.16.79.2:36677/ using document root <none> and password file <none>
I20260620 23:13:02.730456 20470 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:02.730543 20470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:02.730708 20470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:02.732026 20470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "3c4de60cd1744ada890e3275929140bd"
format_stamp: "Formatted at 2026-06-20 23:13:02 on dist-test-slave-pswb"
server_key: "f85565fc4795be87f84eacf3b0057278"
server_key_iv: "e7f4edc79a11bcf068da529ba0d72682"
server_key_version: "encryptionkey@0"
I20260620 23:13:02.732435 20470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "3c4de60cd1744ada890e3275929140bd"
format_stamp: "Formatted at 2026-06-20 23:13:02 on dist-test-slave-pswb"
server_key: "f85565fc4795be87f84eacf3b0057278"
server_key_iv: "e7f4edc79a11bcf068da529ba0d72682"
server_key_version: "encryptionkey@0"
I20260620 23:13:02.734328 20470 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 23:13:02.735297 20485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:02.735554 20470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:02.735697 20470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "3c4de60cd1744ada890e3275929140bd"
format_stamp: "Formatted at 2026-06-20 23:13:02 on dist-test-slave-pswb"
server_key: "f85565fc4795be87f84eacf3b0057278"
server_key_iv: "e7f4edc79a11bcf068da529ba0d72682"
server_key_version: "encryptionkey@0"
I20260620 23:13:02.735811 20470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:02.756446 20470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:02.758492 20470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:02.758757 20470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:02.758991 20470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:02.759302 20470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:02.759358 20470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:02.759397 20470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:02.759425 20470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:02.765209 20470 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:41675
I20260620 23:13:02.765255 20598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:41675 every 8 connection(s)
I20260620 23:13:02.765543 20470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:13:02.770606 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 20470
I20260620 23:13:02.770749 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:02.770951 16700 external_mini_cluster.cc:1428] Setting key d27f4fd66dbf94add26486d99a2f5852
Jun 20 23:13:02 dist-test-slave-pswb krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:02.776830 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:02.766302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:45399 (local address 127.16.79.62:33895)
0620 23:13:02.766355 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:02.766357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:02.766714 (+ 357us) server_negotiation.cc:407] Connection header received
0620 23:13:02.767719 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:02.767721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:02.767736 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:02.767832 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:02.768451 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.769220 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.770378 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.770592 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.770667 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:02.773622 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:02.773624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:02.773625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:02.773632 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:02.775204 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:02.775773 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:02.775774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:02.775774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:02.775830 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:02.776123 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:02.776124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:02.776124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:02.776238 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:02.776290 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:02.776628 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 23:13:02.776762 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:13:02.777078 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:02.766154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33895 (local address 127.16.79.2:45399)
0620 23:13:02.766521 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:02.766567 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:02.767565 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:02.767902 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:02.767905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:02.768156 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:02.768336 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:02.768339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.769328 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.769328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:02.770261 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:02.770263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.770706 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.770708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:02.770880 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:02.771398 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:02.771408 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:02.773494 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:02.775343 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:02.775349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:02.775352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:02.775680 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:02.775905 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:02.775906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:02.775906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:02.776049 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:02.776306 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:02.776312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:02.776480 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 23:13:02.776603 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 23:13:02.776806 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260620 23:13:02.777544 20599 heartbeater.cc:347] Connected to a master server at 127.16.79.62:33895
I20260620 23:13:02.777655 20599 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:02.777829 20599 heartbeater.cc:511] Master 127.16.79.62:33895 requested a full tablet report, sending...
I20260620 23:13:02.778190 20274 ts_manager.cc:194] Registered new tserver with Master: 3c4de60cd1744ada890e3275929140bd (127.16.79.2:41675)
I20260620 23:13:02.778571 20274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:45399
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:13:02.799589 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:33895
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:02.878214 20604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:02.878408 20604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:02.878430 20604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:02.879592 20604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:02.879632 20604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:02.879659 20604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:02.881165 20604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:33895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:02.881399 20604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:02.881596 20604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:02.886893 20612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:02.886883 20610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:02.886883 20604 server_base.cc:1034] running on GCE node
W20260620 23:13:02.886919 20609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:02.887283 20604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:02.887524 20604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:02.888773 20604 hybrid_clock.cc:648] HybridClock initialized: now 1781997182888741 us; error 45 us; skew 500 ppm
Jun 20 23:13:02 dist-test-slave-pswb krb5kdc[20229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:02.891297 20604 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:13:02.891752 20604 webserver.cc:460] Webserver started at http://127.16.79.3:36395/ using document root <none> and password file <none>
I20260620 23:13:02.891984 20604 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:02.892057 20604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:02.892212 20604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:02.893321 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "1e2c48e8518e4cd79fd5e3cb0ca1ddc2"
format_stamp: "Formatted at 2026-06-20 23:13:02 on dist-test-slave-pswb"
server_key: "35abfcdf88e2f9ec1aafdd7ab745fc40"
server_key_iv: "738f4f97f655b878eb906911d08cb10d"
server_key_version: "encryptionkey@0"
I20260620 23:13:02.893692 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "1e2c48e8518e4cd79fd5e3cb0ca1ddc2"
format_stamp: "Formatted at 2026-06-20 23:13:02 on dist-test-slave-pswb"
server_key: "35abfcdf88e2f9ec1aafdd7ab745fc40"
server_key_iv: "738f4f97f655b878eb906911d08cb10d"
server_key_version: "encryptionkey@0"
I20260620 23:13:02.895243 20604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:13:02.896327 20619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:02.896569 20604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:02.896679 20604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "1e2c48e8518e4cd79fd5e3cb0ca1ddc2"
format_stamp: "Formatted at 2026-06-20 23:13:02 on dist-test-slave-pswb"
server_key: "35abfcdf88e2f9ec1aafdd7ab745fc40"
server_key_iv: "738f4f97f655b878eb906911d08cb10d"
server_key_version: "encryptionkey@0"
I20260620 23:13:02.896812 20604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:02.913899 20604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:02.916608 20604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:02.916811 20604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:02.917060 20604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:02.917366 20604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:02.917402 20604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:02.917425 20604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:02.917438 20604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:02.925232 20604 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:35493
I20260620 23:13:02.925282 20732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:35493 every 8 connection(s)
I20260620 23:13:02.925606 20604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:13:02 dist-test-slave-pswb krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:02.934657 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 20604
I20260620 23:13:02.934768 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:02.934974 16700 external_mini_cluster.cc:1428] Setting key 1f81d6f5a2c8d3c63085f7509d6fd66a
I20260620 23:13:02.935624 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:02.926487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:35023 (local address 127.16.79.62:33895)
0620 23:13:02.926656 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:02.926659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:02.926968 (+ 309us) server_negotiation.cc:407] Connection header received
0620 23:13:02.927970 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:02.927971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:02.927985 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:02.928036 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:02.928644 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.929211 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.930193 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.930440 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.930521 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:02.932580 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:02.932581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:02.932581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:02.932587 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:02.934037 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:02.934486 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:02.934487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:02.934488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:02.934546 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:02.934897 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:02.934898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:02.934899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:02.935040 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:02.935111 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:02.935453 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 23:13:02.935528 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 23:13:02.935808 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:02.926465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33895 (local address 127.16.79.3:35023)
0620 23:13:02.926716 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:02.926772 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:02.927843 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:02.928106 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:02.928109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:02.928357 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:02.928547 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:02.928551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.929315 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.929316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:02.930103 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:02.930106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.930530 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.930531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:02.930647 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:02.931148 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:02.931156 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:02.932487 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:02.934148 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:02.934154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:02.934155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:02.934392 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:02.934630 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:02.934630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:02.934631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:02.934734 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:02.935136 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:02.935142 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:02.935302 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 23:13:02.935417 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 23:13:02.935599 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260620 23:13:02.936221 20733 heartbeater.cc:347] Connected to a master server at 127.16.79.62:33895
I20260620 23:13:02.936324 20733 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:02.936498 20733 heartbeater.cc:511] Master 127.16.79.62:33895 requested a full tablet report, sending...
I20260620 23:13:02.936894 20274 ts_manager.cc:194] Registered new tserver with Master: 1e2c48e8518e4cd79fd5e3cb0ca1ddc2 (127.16.79.3:35493)
I20260620 23:13:02.937268 20274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:35023
I20260620 23:13:02.947150 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:13:02 dist-test-slave-pswb krb5kdc[20229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:13:02 dist-test-slave-pswb krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.1@KRBTEST.COM
I20260620 23:13:02.971033 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:02.962873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.16.79.1:34349)
0620 23:13:02.963192 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:02.963197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:02.963209 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:13:02.963235 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:02.963237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:02.963252 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:02.963340 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:02.963937 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.964672 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.965545 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.965850 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.965919 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:02.968071 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:02.968078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:02.968078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:02.968093 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:02.969700 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:02.970293 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:02.970295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:02.970296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:02.970350 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:02.970583 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:02.970583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:02.970584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:02.970699 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:02.970799 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:02.970867 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 23:13:02.970925 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":216,"threads_started":1}
Jun 20 23:13:02 dist-test-slave-pswb krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:02.979691 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:02.972651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.16.79.62:33895)
0620 23:13:02.972778 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:02.972780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:02.972818 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:13:02.972906 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:02.972907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:02.972916 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:02.972965 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:02.973564 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.974030 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.974703 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:02.974892 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:02.974949 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:02.977034 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:02.977035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:02.977035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:02.977041 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:02.978273 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:02.978792 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:02.978793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:02.978793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:02.978842 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:02.979180 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:02.979181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:02.979181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:02.979336 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:02.979413 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:02.979519 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 23:13:02.979575 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260620 23:13:02.979959 20274 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34260
I20260620 23:13:02.980450 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 20336
I20260620 23:13:02.985260 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 20470
I20260620 23:13:02.990335 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 20604
I20260620 23:13:02.994446 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 20242
2026-06-20T23:13:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1895 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:13:03 dist-test-slave-pswb krb5kdc[20745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:13:03 dist-test-slave-pswb krb5kdc[20745](info): set up 2 sockets
Jun 20 23:13:03 dist-test-slave-pswb krb5kdc[20745](info): commencing operation
krb5kdc: starting...
W20260620 23:13:04.044876 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:13:04 dist-test-slave-pswb krb5kdc[20745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:13:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:13:04.167320 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:34617
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:34617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:04.247925 20758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:04.248185 20758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:04.248229 20758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:04.249408 20758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:04.249483 20758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:04.249509 20758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:04.249537 20758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:04.249570 20758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:04.251053 20758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:34617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:34617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:04.251351 20758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:04.251586 20758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:04.256582 20763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:04.256613 20758 server_base.cc:1034] running on GCE node
W20260620 23:13:04.256664 20764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:04.256582 20766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:04.257020 20758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:04.257330 20758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:04.258507 20758 hybrid_clock.cc:648] HybridClock initialized: now 1781997184258509 us; error 45 us; skew 500 ppm
Jun 20 23:13:04 dist-test-slave-pswb krb5kdc[20745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:04.265930 20758 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:13:04.266446 20758 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:04.266531 20758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:04.266700 20758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:04.268018 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "7a70729a61f04e0ba43343eb6ada3165"
format_stamp: "Formatted at 2026-06-20 23:13:04 on dist-test-slave-pswb"
server_key: "31f1a3f889205bea41d226487f417631"
server_key_iv: "41cb83ca0896bebd212b10be177b5082"
server_key_version: "encryptionkey@0"
I20260620 23:13:04.268442 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "7a70729a61f04e0ba43343eb6ada3165"
format_stamp: "Formatted at 2026-06-20 23:13:04 on dist-test-slave-pswb"
server_key: "31f1a3f889205bea41d226487f417631"
server_key_iv: "41cb83ca0896bebd212b10be177b5082"
server_key_version: "encryptionkey@0"
I20260620 23:13:04.270099 20758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 23:13:04.271171 20772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:04.271514 20758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:04.271682 20758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "7a70729a61f04e0ba43343eb6ada3165"
format_stamp: "Formatted at 2026-06-20 23:13:04 on dist-test-slave-pswb"
server_key: "31f1a3f889205bea41d226487f417631"
server_key_iv: "41cb83ca0896bebd212b10be177b5082"
server_key_version: "encryptionkey@0"
I20260620 23:13:04.271817 20758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:04.284247 20758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:04.286475 20758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:04.286638 20758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:04.292033 20758 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:34617
I20260620 23:13:04.292061 20824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:34617 every 8 connection(s)
I20260620 23:13:04.292443 20758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:04.293555 20825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:04.295866 20825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165: Bootstrap starting.
I20260620 23:13:04.296466 20825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:04.296793 20825 log.cc:824] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:04.297504 20825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165: No bootstrap required, opened a new log
I20260620 23:13:04.299017 20825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a70729a61f04e0ba43343eb6ada3165" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34617 } }
I20260620 23:13:04.299151 20825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:04.299173 20825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a70729a61f04e0ba43343eb6ada3165, State: Initialized, Role: FOLLOWER
I20260620 23:13:04.299331 20825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a70729a61f04e0ba43343eb6ada3165" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34617 } }
I20260620 23:13:04.299403 20825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:04.299445 20825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:04.299494 20825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:04.300276 20825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a70729a61f04e0ba43343eb6ada3165" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34617 } }
I20260620 23:13:04.300361 20825 leader_election.cc:302] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a70729a61f04e0ba43343eb6ada3165; no voters:
I20260620 23:13:04.300525 20825 leader_election.cc:288] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:04.300678 20828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:04.300840 20825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:04.300993 20828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [term 1 LEADER]: Becoming Leader. State: Replica: 7a70729a61f04e0ba43343eb6ada3165, State: Running, Role: LEADER
I20260620 23:13:04.301201 20828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a70729a61f04e0ba43343eb6ada3165" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34617 } }
I20260620 23:13:04.301844 20830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a70729a61f04e0ba43343eb6ada3165. Latest consensus state: current_term: 1 leader_uuid: "7a70729a61f04e0ba43343eb6ada3165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a70729a61f04e0ba43343eb6ada3165" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34617 } } }
I20260620 23:13:04.301961 20830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:04.301910 20829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a70729a61f04e0ba43343eb6ada3165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a70729a61f04e0ba43343eb6ada3165" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34617 } } }
I20260620 23:13:04.302022 20829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:04.302297 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 20758
I20260620 23:13:04.302408 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:04.302417 20837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:04.302601 16700 external_mini_cluster.cc:1428] Setting key 1bdb89d2a30a71c06bf80c62556b5c1b
I20260620 23:13:04.302950 20837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:04.305089 20837 catalog_manager.cc:1347] Generated new cluster ID: 681c61e85c01487ca8359f13b17c4b49
I20260620 23:13:04.305239 20837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:13:04 dist-test-slave-pswb krb5kdc[20745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:04.316113 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.303360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.16.79.62:34617)
0620 23:13:04.303823 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:04.303835 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:04.303871 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:13:04.304025 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:04.304034 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:04.304100 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:04.304244 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:04.305356 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.307069 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.308304 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.308968 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.309070 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.312373 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:04.312378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:04.312378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:04.312391 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:04.314437 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:04.314831 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:04.314834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:04.314835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:04.314938 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:04.315139 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:04.315140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:04.315140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:04.315503 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:04.315667 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:04.315733 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 23:13:04.315878 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":86,"threads_started":1}
I20260620 23:13:04.325269 20837 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:04.325994 20837 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:04.328401 20837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165: Generated new TSK 0
I20260620 23:13:04.328933 20837 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:13:04.365463 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34617
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:04.461519 20851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:04.461705 20851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:04.461728 20851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:04.462836 20851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:04.462875 20851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:04.462901 20851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:04.464323 20851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:04.464547 20851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:04.464818 20851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:04.469784 20859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:04.469794 20856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:04.469797 20857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:04.470177 20851 server_base.cc:1034] running on GCE node
I20260620 23:13:04.470407 20851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:04.470638 20851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:04.471781 20851 hybrid_clock.cc:648] HybridClock initialized: now 1781997184471759 us; error 38 us; skew 500 ppm
Jun 20 23:13:04 dist-test-slave-pswb krb5kdc[20745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:04.474313 20851 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:13:04.474717 20851 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:04.474766 20851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:04.474903 20851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:04.476002 20851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "efb4a7b94c9a4ba7a8e7aac082345967"
format_stamp: "Formatted at 2026-06-20 23:13:04 on dist-test-slave-pswb"
server_key: "cfd960ba318d9053899d7b0ad144b7f1"
server_key_iv: "d84281ca5c744ba99c494d9324046c07"
server_key_version: "encryptionkey@0"
I20260620 23:13:04.476351 20851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "efb4a7b94c9a4ba7a8e7aac082345967"
format_stamp: "Formatted at 2026-06-20 23:13:04 on dist-test-slave-pswb"
server_key: "cfd960ba318d9053899d7b0ad144b7f1"
server_key_iv: "d84281ca5c744ba99c494d9324046c07"
server_key_version: "encryptionkey@0"
I20260620 23:13:04.477759 20851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:13:04.478561 20865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:04.478762 20851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:04.478865 20851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "efb4a7b94c9a4ba7a8e7aac082345967"
format_stamp: "Formatted at 2026-06-20 23:13:04 on dist-test-slave-pswb"
server_key: "cfd960ba318d9053899d7b0ad144b7f1"
server_key_iv: "d84281ca5c744ba99c494d9324046c07"
server_key_version: "encryptionkey@0"
I20260620 23:13:04.478945 20851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:04.494848 20851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:04.497180 20851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:04.497339 20851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:04.497579 20851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:04.497884 20851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:04.497938 20851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:04.497978 20851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:04.498006 20851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:04.505031 20851 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:39433
I20260620 23:13:04.505075 20978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:39433 every 8 connection(s)
I20260620 23:13:04.505329 20851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:04.511087 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 20851
I20260620 23:13:04.511226 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
Jun 20 23:13:04 dist-test-slave-pswb krb5kdc[20745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:04.511448 16700 external_mini_cluster.cc:1428] Setting key e5f34a901ba7ba79a3b75120fb6e9ddb
I20260620 23:13:04.515189 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.505997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:58083 (local address 127.16.79.62:34617)
0620 23:13:04.506060 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:04.506062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:04.506229 (+ 167us) server_negotiation.cc:407] Connection header received
0620 23:13:04.506746 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:04.506747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:04.506760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:04.506824 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:04.507717 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.508209 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.509228 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.509422 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.509506 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.512006 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:04.512008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:04.512009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:04.512017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:04.513714 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:04.514176 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:04.514178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:04.514178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:04.514240 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:04.514514 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:04.514515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:04.514515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:04.514638 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:04.514719 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:04.515006 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 23:13:04.515092 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:13:04.515393 20981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.505950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34617 (local address 127.16.79.1:58083)
0620 23:13:04.506092 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:04.506141 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:04.506631 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:04.506906 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:04.506910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:04.507149 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:04.507617 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.507621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.508310 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.508312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.509119 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.509121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.509514 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.509515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.509633 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.510074 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:04.510081 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:04.511904 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:04.513835 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:04.513842 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:04.513844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:04.514078 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:04.514321 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:04.514322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:04.514322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:04.514429 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:04.514709 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:04.514715 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:04.514865 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 23:13:04.514982 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:13:04.515140 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260620 23:13:04.515923 20979 heartbeater.cc:347] Connected to a master server at 127.16.79.62:34617
I20260620 23:13:04.516011 20979 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:04.516156 20979 heartbeater.cc:511] Master 127.16.79.62:34617 requested a full tablet report, sending...
I20260620 23:13:04.516671 20789 ts_manager.cc:194] Registered new tserver with Master: efb4a7b94c9a4ba7a8e7aac082345967 (127.16.79.1:39433)
I20260620 23:13:04.517292 20789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:58083
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:13:04.541061 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34617
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:04.625733 20984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:04.625942 20984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:04.625976 20984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:04.627084 20984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:04.627143 20984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:04.627187 20984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:04.628821 20984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:04.629057 20984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:04.629276 20984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:04.633970 20989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:04.634174 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:04.634248 20984 server_base.cc:1034] running on GCE node
W20260620 23:13:04.634174 20992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:04.634737 20984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:04.635041 20984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:04.636409 20984 hybrid_clock.cc:648] HybridClock initialized: now 1781997184636229 us; error 40 us; skew 500 ppm
Jun 20 23:13:04 dist-test-slave-pswb krb5kdc[20745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:04.639492 20984 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:13:04.639881 20984 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:04.639950 20984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:04.640086 20984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:04.641141 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "6d562e82a9b845c79d1228535dfa0fd3"
format_stamp: "Formatted at 2026-06-20 23:13:04 on dist-test-slave-pswb"
server_key: "2551dda0a1c92dc0d1b3692643ad3d0a"
server_key_iv: "6884d722a6e7e399c9d08b95f410b6c5"
server_key_version: "encryptionkey@0"
I20260620 23:13:04.641479 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "6d562e82a9b845c79d1228535dfa0fd3"
format_stamp: "Formatted at 2026-06-20 23:13:04 on dist-test-slave-pswb"
server_key: "2551dda0a1c92dc0d1b3692643ad3d0a"
server_key_iv: "6884d722a6e7e399c9d08b95f410b6c5"
server_key_version: "encryptionkey@0"
I20260620 23:13:04.642752 20984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:04.643476 20998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:04.643687 20984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:04.643795 20984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "6d562e82a9b845c79d1228535dfa0fd3"
format_stamp: "Formatted at 2026-06-20 23:13:04 on dist-test-slave-pswb"
server_key: "2551dda0a1c92dc0d1b3692643ad3d0a"
server_key_iv: "6884d722a6e7e399c9d08b95f410b6c5"
server_key_version: "encryptionkey@0"
I20260620 23:13:04.643875 20984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:04.661391 20984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:04.664055 20984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:04.664191 20984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:04.664392 20984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:04.664732 20984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:04.664775 20984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:04.664826 20984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:04.664855 20984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:04.670854 20984 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:33041
I20260620 23:13:04.670892 21111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:33041 every 8 connection(s)
I20260620 23:13:04.671132 20984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:13:04.676611 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 20984
I20260620 23:13:04.676744 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:04.676954 16700 external_mini_cluster.cc:1428] Setting key 0f7bf78a8be307eafb99430c69871720
Jun 20 23:13:04 dist-test-slave-pswb krb5kdc[20745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:04.681439 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.671748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34617 (local address 127.16.79.2:38599)
0620 23:13:04.671930 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:04.671963 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:04.672413 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:04.672601 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:04.672604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:04.672961 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:04.673432 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.673437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.674178 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.674179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.675027 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.675030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.675427 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.675428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.675552 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.676034 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:04.676041 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:04.678187 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:04.680046 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:04.680051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:04.680053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:04.680283 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:04.680516 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:04.680517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:04.680517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:04.680622 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:04.680959 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:04.680965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:04.681076 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 23:13:04.681171 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 23:13:04.681290 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
I20260620 23:13:04.681437 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.671765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:38599 (local address 127.16.79.62:34617)
0620 23:13:04.671848 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:04.671851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:04.672046 (+ 195us) server_negotiation.cc:407] Connection header received
0620 23:13:04.672491 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:04.672492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:04.672505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:04.672555 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:04.673506 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.673991 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.675124 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.675332 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.675419 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.678298 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:04.678300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:04.678300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:04.678306 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:04.679942 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:04.680365 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:04.680366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:04.680367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:04.680426 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:04.680727 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:04.680728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:04.680729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:04.680869 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:04.680940 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:04.681210 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:13:04.681297 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:13:04.682328 21112 heartbeater.cc:347] Connected to a master server at 127.16.79.62:34617
I20260620 23:13:04.682435 21112 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:04.682593 21112 heartbeater.cc:511] Master 127.16.79.62:34617 requested a full tablet report, sending...
I20260620 23:13:04.682960 20789 ts_manager.cc:194] Registered new tserver with Master: 6d562e82a9b845c79d1228535dfa0fd3 (127.16.79.2:33041)
I20260620 23:13:04.683491 20789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:38599
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:13:04.704473 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34617
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:04.781888 21117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:04.782104 21117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:04.782132 21117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:04.783248 21117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:04.783288 21117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:04.783314 21117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:04.784792 21117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:04.784999 21117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:04.785211 21117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:04.790051 21123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:04.790088 21125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:04.790207 21117 server_base.cc:1034] running on GCE node
W20260620 23:13:04.790162 21122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:04.790505 21117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:04.790736 21117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:04.791896 21117 hybrid_clock.cc:648] HybridClock initialized: now 1781997184791872 us; error 38 us; skew 500 ppm
Jun 20 23:13:04 dist-test-slave-pswb krb5kdc[20745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:04.794895 21117 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:13:04.795264 21117 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:04.795334 21117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:04.795469 21117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:04.796466 21117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "b3686850304b4f0b97b070a8e67910f4"
format_stamp: "Formatted at 2026-06-20 23:13:04 on dist-test-slave-pswb"
server_key: "5190f5258244ab9efcfbcd3648d572c8"
server_key_iv: "3078fb9fda071e8d30c60e385b81402f"
server_key_version: "encryptionkey@0"
I20260620 23:13:04.796844 21117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "b3686850304b4f0b97b070a8e67910f4"
format_stamp: "Formatted at 2026-06-20 23:13:04 on dist-test-slave-pswb"
server_key: "5190f5258244ab9efcfbcd3648d572c8"
server_key_iv: "3078fb9fda071e8d30c60e385b81402f"
server_key_version: "encryptionkey@0"
I20260620 23:13:04.798421 21117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:04.799273 21131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:04.799474 21117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:04.799569 21117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "b3686850304b4f0b97b070a8e67910f4"
format_stamp: "Formatted at 2026-06-20 23:13:04 on dist-test-slave-pswb"
server_key: "5190f5258244ab9efcfbcd3648d572c8"
server_key_iv: "3078fb9fda071e8d30c60e385b81402f"
server_key_version: "encryptionkey@0"
I20260620 23:13:04.799655 21117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:04.827096 21117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:04.829720 21117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:04.829901 21117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:04.830140 21117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:04.830469 21117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:04.830524 21117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:04.830559 21117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:04.830586 21117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:04.836220 21117 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:44963
I20260620 23:13:04.836514 21117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:13:04.836537 21244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:44963 every 8 connection(s)
I20260620 23:13:04.838908 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 21117
I20260620 23:13:04.839008 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:04.839191 16700 external_mini_cluster.cc:1428] Setting key 7bbadf0fa86e81b4d6d1e71c62ff58e2
Jun 20 23:13:04 dist-test-slave-pswb krb5kdc[20745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:04.846413 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.837176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:58699 (local address 127.16.79.62:34617)
0620 23:13:04.837238 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:04.837241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:04.837453 (+ 212us) server_negotiation.cc:407] Connection header received
0620 23:13:04.837906 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:04.837907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:04.837917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:04.837973 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:04.838887 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.839608 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.840776 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.840966 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.841029 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.843159 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:04.843160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:04.843160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:04.843166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:04.844570 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:04.845385 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:04.845386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:04.845386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:04.845437 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:04.845735 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:04.845736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:04.845736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:04.845852 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:04.845921 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:04.846272 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 23:13:04.846334 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:13:04.846642 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.837196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34617 (local address 127.16.79.3:58699)
0620 23:13:04.837333 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:04.837367 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:04.837802 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:04.838019 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:04.838022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:04.838274 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:04.838771 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.838776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.839689 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.839691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.840653 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.840655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.841043 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.841043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.841132 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.841580 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:04.841587 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:04.843074 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:04.844653 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:04.844673 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:04.844675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:04.845267 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:04.845540 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:04.845541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:04.845541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:04.845649 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:04.845961 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:04.845966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:04.846076 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 23:13:04.846217 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 23:13:04.846401 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":82,"threads_started":1}
I20260620 23:13:04.847151 21245 heartbeater.cc:347] Connected to a master server at 127.16.79.62:34617
I20260620 23:13:04.847370 21245 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:04.847556 21245 heartbeater.cc:511] Master 127.16.79.62:34617 requested a full tablet report, sending...
I20260620 23:13:04.848122 20789 ts_manager.cc:194] Registered new tserver with Master: b3686850304b4f0b97b070a8e67910f4 (127.16.79.3:44963)
I20260620 23:13:04.848479 20789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:58699
I20260620 23:13:04.851096 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:13:04.858409 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.852336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.16.79.62:34617)
0620 23:13:04.852396 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:04.852398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:04.852503 (+ 105us) server_negotiation.cc:407] Connection header received
0620 23:13:04.852542 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:04.852543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:04.852550 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:04.852605 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:04.853240 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.853689 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.854401 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.854594 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.854658 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.855765 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:04.855766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:04.855767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:04.855772 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:04.857338 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:04.857801 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:04.857802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:04.857802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:04.857848 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:04.858021 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:04.858021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:04.858022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:04.858144 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:04.858215 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:04.858308 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 23:13:04.858357 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:13:04.859931 20789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:13:04.861500 20789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:13:04.868532 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.863887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:44963 (local address 127.0.0.1:40470)
0620 23:13:04.864344 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:04.864362 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:04.864473 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:04.864824 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:04.864825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:04.864828 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:04.864904 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.864906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.866119 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.866120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.867544 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.867545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.868333 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.868334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.868396 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.868400 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:04.868444 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 23:13:04.868476 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":44,"thread_start_us":61,"threads_started":1}
I20260620 23:13:04.868680 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.863798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.16.79.1:39433)
0620 23:13:04.864239 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:04.864242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:04.864252 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:13:04.864291 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:04.864293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:04.864329 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:04.864407 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:04.864665 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.865750 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.867621 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.868308 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.868398 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.868446 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:04.868552 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 23:13:04.868587 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":76,"threads_started":1}
I20260620 23:13:04.868716 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.863861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.16.79.2:33041)
0620 23:13:04.864044 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:04.864048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:04.864224 (+ 176us) server_negotiation.cc:407] Connection header received
0620 23:13:04.864333 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:04.864337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:04.864363 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:04.864680 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:04.865021 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.866000 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.867448 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.868133 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.868209 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.868265 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:04.868575 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 23:13:04.868613 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260620 23:13:04.868531 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.863692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:33041 (local address 127.0.0.1:37088)
0620 23:13:04.864040 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:04.864050 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:04.864118 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:04.864852 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:04.864853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:04.864855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:04.864914 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.864916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.866110 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.866111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.867359 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.867362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.868237 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.868238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.868314 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.868329 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 23:13:04.868396 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:13:04.868455 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":56,"thread_start_us":44,"threads_started":1}
I20260620 23:13:04.868847 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.864021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.16.79.3:44963)
0620 23:13:04.864180 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:04.864185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:04.864558 (+ 373us) server_negotiation.cc:407] Connection header received
0620 23:13:04.864612 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:04.864615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:04.864635 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:04.864747 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:04.865123 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.865992 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.867651 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.868144 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.868236 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.868300 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:04.868684 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 23:13:04.868760 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260620 23:13:04.869515 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.863690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:39433 (local address 127.0.0.1:39636)
0620 23:13:04.863828 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:04.863864 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:04.863931 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:04.864492 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:04.864495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:04.864498 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:04.864582 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.864584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.865880 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.865881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.867529 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.867532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.868400 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.868401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.868477 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.868482 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:13:04.868539 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 23:13:04.869438 (+ 899us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":89,"threads_started":1}
I20260620 23:13:04.869616 21046 tablet_service.cc:1467] Processing CreateTablet for tablet 07e54aafd1c04ed6b328c2712b46d737 (DEFAULT_TABLE table=test-table [id=fb4d0347ab004b62b03d09c979d99eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:04.869987 21046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e54aafd1c04ed6b328c2712b46d737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:04.870081 20913 tablet_service.cc:1467] Processing CreateTablet for tablet 07e54aafd1c04ed6b328c2712b46d737 (DEFAULT_TABLE table=test-table [id=fb4d0347ab004b62b03d09c979d99eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:04.870138 21179 tablet_service.cc:1467] Processing CreateTablet for tablet 07e54aafd1c04ed6b328c2712b46d737 (DEFAULT_TABLE table=test-table [id=fb4d0347ab004b62b03d09c979d99eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:04.870308 20913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e54aafd1c04ed6b328c2712b46d737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:04.870410 21179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e54aafd1c04ed6b328c2712b46d737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:04.872750 21262 tablet_bootstrap.cc:492] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967: Bootstrap starting.
I20260620 23:13:04.873265 21263 tablet_bootstrap.cc:492] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3: Bootstrap starting.
I20260620 23:13:04.873417 21262 tablet_bootstrap.cc:654] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:04.873507 21264 tablet_bootstrap.cc:492] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4: Bootstrap starting.
I20260620 23:13:04.874022 21262 log.cc:824] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:04.874118 21264 tablet_bootstrap.cc:654] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:04.874121 21263 tablet_bootstrap.cc:654] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:04.874670 21263 log.cc:824] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:04.874728 21264 log.cc:824] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:04.874858 21262 tablet_bootstrap.cc:492] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967: No bootstrap required, opened a new log
I20260620 23:13:04.874929 21262 ts_tablet_manager.cc:1397] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:13:04.875377 21263 tablet_bootstrap.cc:492] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3: No bootstrap required, opened a new log
I20260620 23:13:04.875391 21264 tablet_bootstrap.cc:492] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4: No bootstrap required, opened a new log
I20260620 23:13:04.875449 21263 ts_tablet_manager.cc:1397] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:13:04.875547 21264 ts_tablet_manager.cc:1397] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:13:04.876668 21262 raft_consensus.cc:348] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb4a7b94c9a4ba7a8e7aac082345967" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39433 } } peers { permanent_uuid: "b3686850304b4f0b97b070a8e67910f4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44963 } } peers { permanent_uuid: "6d562e82a9b845c79d1228535dfa0fd3" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33041 } }
I20260620 23:13:04.876895 21262 raft_consensus.cc:374] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:04.876940 21262 raft_consensus.cc:729] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb4a7b94c9a4ba7a8e7aac082345967, State: Initialized, Role: FOLLOWER
I20260620 23:13:04.877046 21262 consensus_queue.cc:260] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb4a7b94c9a4ba7a8e7aac082345967" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39433 } } peers { permanent_uuid: "b3686850304b4f0b97b070a8e67910f4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44963 } } peers { permanent_uuid: "6d562e82a9b845c79d1228535dfa0fd3" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33041 } }
I20260620 23:13:04.877323 21262 ts_tablet_manager.cc:1428] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 23:13:04.877457 20979 heartbeater.cc:503] Master 127.16.79.62:34617 was elected leader, sending a full tablet report...
I20260620 23:13:04.877431 21263 raft_consensus.cc:348] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb4a7b94c9a4ba7a8e7aac082345967" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39433 } } peers { permanent_uuid: "b3686850304b4f0b97b070a8e67910f4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44963 } } peers { permanent_uuid: "6d562e82a9b845c79d1228535dfa0fd3" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33041 } }
I20260620 23:13:04.877431 21264 raft_consensus.cc:348] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb4a7b94c9a4ba7a8e7aac082345967" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39433 } } peers { permanent_uuid: "b3686850304b4f0b97b070a8e67910f4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44963 } } peers { permanent_uuid: "6d562e82a9b845c79d1228535dfa0fd3" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33041 } }
I20260620 23:13:04.877580 21263 raft_consensus.cc:374] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:04.877580 21264 raft_consensus.cc:374] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:04.877621 21264 raft_consensus.cc:729] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3686850304b4f0b97b070a8e67910f4, State: Initialized, Role: FOLLOWER
I20260620 23:13:04.877621 21263 raft_consensus.cc:729] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d562e82a9b845c79d1228535dfa0fd3, State: Initialized, Role: FOLLOWER
I20260620 23:13:04.877732 21264 consensus_queue.cc:260] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb4a7b94c9a4ba7a8e7aac082345967" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39433 } } peers { permanent_uuid: "b3686850304b4f0b97b070a8e67910f4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44963 } } peers { permanent_uuid: "6d562e82a9b845c79d1228535dfa0fd3" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33041 } }
I20260620 23:13:04.877856 21263 consensus_queue.cc:260] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb4a7b94c9a4ba7a8e7aac082345967" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39433 } } peers { permanent_uuid: "b3686850304b4f0b97b070a8e67910f4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44963 } } peers { permanent_uuid: "6d562e82a9b845c79d1228535dfa0fd3" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33041 } }
I20260620 23:13:04.878001 21264 ts_tablet_manager.cc:1428] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:13:04.878042 21245 heartbeater.cc:503] Master 127.16.79.62:34617 was elected leader, sending a full tablet report...
I20260620 23:13:04.878146 21263 ts_tablet_manager.cc:1428] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:13:04.878268 21112 heartbeater.cc:503] Master 127.16.79.62:34617 was elected leader, sending a full tablet report...
W20260620 23:13:04.921589 21113 tablet.cc:2369] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:13:04.947239 21270 raft_consensus.cc:482] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:04.947383 21270 raft_consensus.cc:504] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb4a7b94c9a4ba7a8e7aac082345967" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39433 } } peers { permanent_uuid: "b3686850304b4f0b97b070a8e67910f4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44963 } } peers { permanent_uuid: "6d562e82a9b845c79d1228535dfa0fd3" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33041 } }
I20260620 23:13:04.947824 21270 leader_election.cc:288] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb4a7b94c9a4ba7a8e7aac082345967 (127.16.79.1:39433), b3686850304b4f0b97b070a8e67910f4 (127.16.79.3:44963)
I20260620 23:13:04.951435 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.948102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:51265 (local address 127.16.79.1:39433)
0620 23:13:04.948175 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:04.948177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:04.948190 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:13:04.948220 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:04.948221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:04.948234 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:04.948310 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:04.948529 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.949154 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.950449 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.951028 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.951092 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.951101 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:04.951351 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 23:13:04.951383 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:13:04.951457 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.948020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:39433 (local address 127.16.79.2:51265)
0620 23:13:04.948084 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:04.948097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:04.948141 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:04.948336 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:04.948337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:04.948339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:04.948421 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.948423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.949260 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.949262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.950340 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.950343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.951170 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.951171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.951265 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.951270 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:13:04.951334 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:13:04.951381 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":10}
I20260620 23:13:04.951807 20933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e54aafd1c04ed6b328c2712b46d737" candidate_uuid: "6d562e82a9b845c79d1228535dfa0fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb4a7b94c9a4ba7a8e7aac082345967" is_pre_election: true
I20260620 23:13:04.951987 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.948260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:52367 (local address 127.16.79.3:44963)
0620 23:13:04.948322 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:04.948324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:04.948395 (+ 71us) server_negotiation.cc:407] Connection header received
0620 23:13:04.948450 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:04.948451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:04.948464 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:04.948525 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:04.948843 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.949577 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.950924 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.951562 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.951633 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.951643 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:04.951878 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 23:13:04.951921 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:13:04.951977 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.948141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:44963 (local address 127.16.79.2:52367)
0620 23:13:04.948314 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:04.948326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:04.948385 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:04.948540 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:04.948541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:04.948543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:04.948608 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.948610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.949720 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.949721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.950795 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:04.950798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.951705 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.951706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:04.951766 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.951770 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:04.951858 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:13:04.951900 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":118,"threads_started":1}
I20260620 23:13:04.951985 20933 raft_consensus.cc:2454] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d562e82a9b845c79d1228535dfa0fd3 in term 0.
I20260620 23:13:04.952283 21001 leader_election.cc:302] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d562e82a9b845c79d1228535dfa0fd3, efb4a7b94c9a4ba7a8e7aac082345967; no voters:
I20260620 23:13:04.952389 21270 raft_consensus.cc:2781] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:13:04.952461 21270 raft_consensus.cc:482] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:04.952488 21270 raft_consensus.cc:3037] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:04.952620 21199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e54aafd1c04ed6b328c2712b46d737" candidate_uuid: "6d562e82a9b845c79d1228535dfa0fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3686850304b4f0b97b070a8e67910f4" is_pre_election: true
I20260620 23:13:04.952780 21199 raft_consensus.cc:2454] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d562e82a9b845c79d1228535dfa0fd3 in term 0.
I20260620 23:13:04.953356 21270 raft_consensus.cc:504] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb4a7b94c9a4ba7a8e7aac082345967" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39433 } } peers { permanent_uuid: "b3686850304b4f0b97b070a8e67910f4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44963 } } peers { permanent_uuid: "6d562e82a9b845c79d1228535dfa0fd3" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33041 } }
I20260620 23:13:04.953495 21270 leader_election.cc:288] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [CANDIDATE]: Term 1 election: Requested vote from peers efb4a7b94c9a4ba7a8e7aac082345967 (127.16.79.1:39433), b3686850304b4f0b97b070a8e67910f4 (127.16.79.3:44963)
I20260620 23:13:04.953691 21199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e54aafd1c04ed6b328c2712b46d737" candidate_uuid: "6d562e82a9b845c79d1228535dfa0fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3686850304b4f0b97b070a8e67910f4"
I20260620 23:13:04.953769 21199 raft_consensus.cc:3037] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:04.954424 21199 raft_consensus.cc:2454] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d562e82a9b845c79d1228535dfa0fd3 in term 1.
I20260620 23:13:04.954516 20933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e54aafd1c04ed6b328c2712b46d737" candidate_uuid: "6d562e82a9b845c79d1228535dfa0fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb4a7b94c9a4ba7a8e7aac082345967"
I20260620 23:13:04.954600 21001 leader_election.cc:302] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d562e82a9b845c79d1228535dfa0fd3, b3686850304b4f0b97b070a8e67910f4; no voters:
I20260620 23:13:04.954620 20933 raft_consensus.cc:3037] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:04.954715 21270 raft_consensus.cc:2781] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:04.954912 21270 raft_consensus.cc:686] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 6d562e82a9b845c79d1228535dfa0fd3, State: Running, Role: LEADER
I20260620 23:13:04.955032 21270 consensus_queue.cc:237] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb4a7b94c9a4ba7a8e7aac082345967" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39433 } } peers { permanent_uuid: "b3686850304b4f0b97b070a8e67910f4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44963 } } peers { permanent_uuid: "6d562e82a9b845c79d1228535dfa0fd3" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33041 } }
I20260620 23:13:04.955520 20933 raft_consensus.cc:2454] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d562e82a9b845c79d1228535dfa0fd3 in term 1.
I20260620 23:13:04.955875 20788 catalog_manager.cc:5495] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d562e82a9b845c79d1228535dfa0fd3 (127.16.79.2). New cstate: current_term: 1 leader_uuid: "6d562e82a9b845c79d1228535dfa0fd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb4a7b94c9a4ba7a8e7aac082345967" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3686850304b4f0b97b070a8e67910f4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d562e82a9b845c79d1228535dfa0fd3" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 33041 } health_report { overall_health: HEALTHY } } }
I20260620 23:13:04.961082 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.958576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.16.79.2:33041)
0620 23:13:04.958637 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:04.958639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:04.958646 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:04.958730 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:04.958731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:04.958745 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:04.958805 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:13:04.958988 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.959462 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.960302 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:04.960478 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:04.960538 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:04.960806 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:13:04.960899 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:13:04.960961 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:04.961024 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 23:13:04.961040 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:13:04.962647 21199 raft_consensus.cc:1261] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4 [term 1 FOLLOWER]: Refusing update from remote peer 6d562e82a9b845c79d1228535dfa0fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:04.962679 20933 raft_consensus.cc:1261] T 07e54aafd1c04ed6b328c2712b46d737 P efb4a7b94c9a4ba7a8e7aac082345967 [term 1 FOLLOWER]: Refusing update from remote peer 6d562e82a9b845c79d1228535dfa0fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:04.962976 21272 consensus_queue.cc:1035] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3686850304b4f0b97b070a8e67910f4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:04.963076 21270 consensus_queue.cc:1035] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb4a7b94c9a4ba7a8e7aac082345967" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:04.965783 21275 mvcc.cc:205] Tried to move back new op lower bound from 7299060469604548608 to 7299060469576474624. Current Snapshot: MvccSnapshot[applied={T|T < 7299060469604548608}]
I20260620 23:13:04.965925 21276 mvcc.cc:205] Tried to move back new op lower bound from 7299060469604548608 to 7299060469576474624. Current Snapshot: MvccSnapshot[applied={T|T < 7299060469604548608}]
I20260620 23:13:04.966109 21277 mvcc.cc:205] Tried to move back new op lower bound from 7299060469604548608 to 7299060469576474624. Current Snapshot: MvccSnapshot[applied={T|T < 7299060469604548608}]
I20260620 23:13:04.967831 20788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:04.967927 20788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:04.968604 20788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a70729a61f04e0ba43343eb6ada3165: Sending DeleteTablet for 3 replicas of tablet 07e54aafd1c04ed6b328c2712b46d737
I20260620 23:13:04.968971 21046 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e54aafd1c04ed6b328c2712b46d737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:04 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:37088
I20260620 23:13:04.968977 20913 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e54aafd1c04ed6b328c2712b46d737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:04 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:39636
I20260620 23:13:04.968976 21179 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e54aafd1c04ed6b328c2712b46d737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:04 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:40470
I20260620 23:13:04.969177 21286 tablet_replica.cc:330] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4: stopping tablet replica
I20260620 23:13:04.969261 21286 raft_consensus.cc:2229] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:04.969364 21286 raft_consensus.cc:2258] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:04.969522 21284 tablet_replica.cc:330] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3: stopping tablet replica
I20260620 23:13:04.969518 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 20851
I20260620 23:13:04.969604 21284 raft_consensus.cc:2229] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:13:04.970702 21286 ts_tablet_manager.cc:1905] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:04.972613 21286 ts_tablet_manager.cc:1918] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:04.972708 21286 log.cc:1196] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/07e54aafd1c04ed6b328c2712b46d737
I20260620 23:13:04.973001 21286 ts_tablet_manager.cc:1939] T 07e54aafd1c04ed6b328c2712b46d737 P b3686850304b4f0b97b070a8e67910f4: Deleting consensus metadata
I20260620 23:13:04.973369 20775 catalog_manager.cc:4841] TS b3686850304b4f0b97b070a8e67910f4 (127.16.79.3:44963): tablet 07e54aafd1c04ed6b328c2712b46d737 (table test-table [id=fb4d0347ab004b62b03d09c979d99eb2]) successfully deleted
I20260620 23:13:04.973567 21284 raft_consensus.cc:2258] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:04.974028 21284 ts_tablet_manager.cc:1905] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:04.976053 21284 ts_tablet_manager.cc:1918] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:04.976122 21284 log.cc:1196] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/07e54aafd1c04ed6b328c2712b46d737
I20260620 23:13:04.976377 21284 ts_tablet_manager.cc:1939] T 07e54aafd1c04ed6b328c2712b46d737 P 6d562e82a9b845c79d1228535dfa0fd3: Deleting consensus metadata
I20260620 23:13:04.976775 20773 catalog_manager.cc:4841] TS 6d562e82a9b845c79d1228535dfa0fd3 (127.16.79.2:33041): tablet 07e54aafd1c04ed6b328c2712b46d737 (table test-table [id=fb4d0347ab004b62b03d09c979d99eb2]) successfully deleted
W20260620 23:13:04.976938 20775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:39433 (error 108)
I20260620 23:13:04.977416 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:04.977306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:39433 (local address 127.0.0.1:39640)
0620 23:13:04.977351 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:04.977381 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 23:13:04.977489 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 20984
W20260620 23:13:04.977506 20775 catalog_manager.cc:4584] TS efb4a7b94c9a4ba7a8e7aac082345967 (127.16.79.1:39433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07e54aafd1c04ed6b328c2712b46d737: Network error: Client connection negotiation failed: client connection to 127.16.79.1:39433: connect: Connection refused (error 111)
I20260620 23:13:04.982687 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 21117
I20260620 23:13:04.986915 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 20758
2026-06-20T23:13:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1991 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T23:13:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:05Z Disabled control of system clock
I20260620 23:13:05.014655 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45005
--webserver_interface=127.16.79.62
--webserver_port=0
--builtin_ntp_servers=127.16.79.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:45005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:13:05.095471 21291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:05.095782 21291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:05.095810 21291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:05.097867 21291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:05.097942 21291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:05.097956 21291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:05.097971 21291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:05.097985 21291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:05.099681 21291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:45005
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:05.099876 21291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:05.100098 21291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:05.104785 21299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:05.104801 21296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:05.104828 21291 server_base.cc:1034] running on GCE node
W20260620 23:13:05.104802 21297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:05.105289 21291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:05.105573 21291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:05.106738 21291 hybrid_clock.cc:648] HybridClock initialized: now 1781997185106721 us; error 43 us; skew 500 ppm
I20260620 23:13:05.107961 21291 webserver.cc:460] Webserver started at http://127.16.79.62:33037/ using document root <none> and password file <none>
I20260620 23:13:05.108183 21291 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:05.108248 21291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:05.108409 21291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:05.109455 21291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "ed301cbc00cf48b1b526a4cf24f43707"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "28e625998e14b1eb6bba2d89316f86c9"
server_key_iv: "d311a2202ceaac8c3ac80ecd5416b7e0"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.109798 21291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "ed301cbc00cf48b1b526a4cf24f43707"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "28e625998e14b1eb6bba2d89316f86c9"
server_key_iv: "d311a2202ceaac8c3ac80ecd5416b7e0"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.111057 21291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:13:05.112049 21305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:05.112345 21291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:05.112459 21291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "ed301cbc00cf48b1b526a4cf24f43707"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "28e625998e14b1eb6bba2d89316f86c9"
server_key_iv: "d311a2202ceaac8c3ac80ecd5416b7e0"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.112555 21291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:05.115834 21291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:05.116119 21291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:05.116274 21291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:05.121245 21291 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:45005
I20260620 23:13:05.121279 21357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:45005 every 8 connection(s)
I20260620 23:13:05.121588 21291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:05.122217 21358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:05.124850 21358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707: Bootstrap starting.
I20260620 23:13:05.125391 21358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:05.125684 21358 log.cc:824] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:05.126585 21358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707: No bootstrap required, opened a new log
I20260620 23:13:05.128101 21358 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed301cbc00cf48b1b526a4cf24f43707" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45005 } }
I20260620 23:13:05.128228 21358 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:05.128269 21358 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed301cbc00cf48b1b526a4cf24f43707, State: Initialized, Role: FOLLOWER
I20260620 23:13:05.128423 21358 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed301cbc00cf48b1b526a4cf24f43707" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45005 } }
I20260620 23:13:05.128499 21358 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:05.128520 21358 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:05.128569 21358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:05.129079 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 21291
I20260620 23:13:05.129122 21358 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed301cbc00cf48b1b526a4cf24f43707" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45005 } }
I20260620 23:13:05.129186 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:05.129225 21358 leader_election.cc:302] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed301cbc00cf48b1b526a4cf24f43707; no voters:
I20260620 23:13:05.129403 16700 external_mini_cluster.cc:1428] Setting key 02cc0fb3a43e9bc1419007a31b45ace3
I20260620 23:13:05.129429 21358 leader_election.cc:288] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:05.129456 21361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:05.129618 21361 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [term 1 LEADER]: Becoming Leader. State: Replica: ed301cbc00cf48b1b526a4cf24f43707, State: Running, Role: LEADER
I20260620 23:13:05.129669 21358 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:05.129765 21361 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed301cbc00cf48b1b526a4cf24f43707" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45005 } }
I20260620 23:13:05.130164 21363 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed301cbc00cf48b1b526a4cf24f43707. Latest consensus state: current_term: 1 leader_uuid: "ed301cbc00cf48b1b526a4cf24f43707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed301cbc00cf48b1b526a4cf24f43707" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45005 } } }
I20260620 23:13:05.130291 21363 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:05.130365 21362 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed301cbc00cf48b1b526a4cf24f43707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed301cbc00cf48b1b526a4cf24f43707" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45005 } } }
I20260620 23:13:05.130450 21362 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707 [sys.catalog]: This master's current role is: LEADER
W20260620 23:13:05.131587 21379 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:13:05.131654 21379 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:13:05.131717 21370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:05.131906 21370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:05.132398 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.129988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.16.79.62:45005)
0620 23:13:05.130687 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.130694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.130747 (+ 53us) server_negotiation.cc:407] Connection header received
0620 23:13:05.130859 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.130866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.130906 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.131047 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.131935 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.131938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.131938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:05.131943 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.131988 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:05.132004 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.132087 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.132161 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.132212 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":80,"threads_started":1}
I20260620 23:13:05.133389 21370 catalog_manager.cc:1347] Generated new cluster ID: 00f8657462554f2695657d937ae55df8
I20260620 23:13:05.133446 21370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:05.141149 21370 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:05.141299 21370 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:05.147938 21370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707: Generated new TSK 0
I20260620 23:13:05.148094 21370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:13:05.153489 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45005
--builtin_ntp_servers=127.16.79.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:13:05.232772 21382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:05.232961 21382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:05.232982 21382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:05.234122 21382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:05.234165 21382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:05.234190 21382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:05.235482 21382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:05.235714 21382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:05.236034 21382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:05.240803 21387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:05.240809 21390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:05.240809 21388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:05.241111 21382 server_base.cc:1034] running on GCE node
I20260620 23:13:05.241299 21382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:05.241571 21382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:05.242743 21382 hybrid_clock.cc:648] HybridClock initialized: now 1781997185242723 us; error 35 us; skew 500 ppm
I20260620 23:13:05.244364 21382 webserver.cc:460] Webserver started at http://127.16.79.1:38431/ using document root <none> and password file <none>
I20260620 23:13:05.244630 21382 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:05.244722 21382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:05.244880 21382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:05.245885 21382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "1a24dfb4057b4f48b71b21bbcb075b08"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "e128ecef561d2a271a9e486b0ec696b8"
server_key_iv: "12dddd4356bcdc631103ea72db029745"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.246227 21382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "1a24dfb4057b4f48b71b21bbcb075b08"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "e128ecef561d2a271a9e486b0ec696b8"
server_key_iv: "12dddd4356bcdc631103ea72db029745"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.247498 21382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:13:05.248260 21396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:05.248457 21382 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:05.248551 21382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "1a24dfb4057b4f48b71b21bbcb075b08"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "e128ecef561d2a271a9e486b0ec696b8"
server_key_iv: "12dddd4356bcdc631103ea72db029745"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.248636 21382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:05.251812 21382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:05.252066 21382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:05.252198 21382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:05.252408 21382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:05.252722 21382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:05.252765 21382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:05.252802 21382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:05.252828 21382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:05.258806 21382 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:37333
I20260620 23:13:05.258844 21509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:37333 every 8 connection(s)
I20260620 23:13:05.259131 21382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:05.262108 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.259775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:47921 (local address 127.16.79.62:45005)
0620 23:13:05.259819 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.259822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.260046 (+ 224us) server_negotiation.cc:407] Connection header received
0620 23:13:05.260472 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.260473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.260479 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.260555 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.261839 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.261841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.261842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:05.261848 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.261869 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:05.261875 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.261946 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.262021 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.262038 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:13:05.262296 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.259736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45005 (local address 127.16.79.1:47921)
0620 23:13:05.259921 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:05.259961 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:05.260386 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:05.260613 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:05.260616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:05.260624 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:05.261675 (+ 1051us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:05.261675 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:05.261701 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:05.261727 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:05.261950 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:05.261955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:05.261957 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:13:05.262024 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:13:05.262069 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260620 23:13:05.262760 21510 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45005
I20260620 23:13:05.262856 21510 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:05.262986 21510 heartbeater.cc:511] Master 127.16.79.62:45005 requested a full tablet report, sending...
I20260620 23:13:05.263430 21322 ts_manager.cc:194] Registered new tserver with Master: 1a24dfb4057b4f48b71b21bbcb075b08 (127.16.79.1:37333)
I20260620 23:13:05.268626 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 21382
I20260620 23:13:05.268767 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:13:05.268936 16700 external_mini_cluster.cc:1428] Setting key cb02c6c57c37000d30b4624124ecbc92
I20260620 23:13:05.269894 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45005
--builtin_ntp_servers=127.16.79.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:13:05.348793 21513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:05.348977 21513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:05.348999 21513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:05.350121 21513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:05.350159 21513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:05.350185 21513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:05.352056 21513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:05.352490 21513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:05.352931 21513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:05.358980 21518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:05.358986 21519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:05.358986 21521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:05.359055 21513 server_base.cc:1034] running on GCE node
I20260620 23:13:05.359426 21513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:05.359634 21513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:05.360777 21513 hybrid_clock.cc:648] HybridClock initialized: now 1781997185360755 us; error 32 us; skew 500 ppm
I20260620 23:13:05.362011 21513 webserver.cc:460] Webserver started at http://127.16.79.2:43549/ using document root <none> and password file <none>
I20260620 23:13:05.362228 21513 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:05.362285 21513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:05.362411 21513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:05.363471 21513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "d74e12f8a00d409e983d01305ba5fd1c"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "4023fbebc7d635351083dae756d08c34"
server_key_iv: "fe79802f5c0324b227054387b4105903"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.363798 21513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "d74e12f8a00d409e983d01305ba5fd1c"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "4023fbebc7d635351083dae756d08c34"
server_key_iv: "fe79802f5c0324b227054387b4105903"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.365224 21513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:13:05.365972 21527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:05.366138 21513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:05.366235 21513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "d74e12f8a00d409e983d01305ba5fd1c"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "4023fbebc7d635351083dae756d08c34"
server_key_iv: "fe79802f5c0324b227054387b4105903"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.366307 21513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:05.369521 21513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:05.369776 21513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:05.369900 21513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:05.370110 21513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:05.370393 21513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:05.370448 21513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:05.370481 21513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:05.370508 21513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:05.376346 21513 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:37697
I20260620 23:13:05.376390 21640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:37697 every 8 connection(s)
I20260620 23:13:05.376740 21513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:13:05.379735 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.377410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:40491 (local address 127.16.79.62:45005)
0620 23:13:05.377461 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.377464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.377662 (+ 198us) server_negotiation.cc:407] Connection header received
0620 23:13:05.378156 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.378157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.378163 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.378240 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.379442 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.379443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.379443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:05.379448 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.379465 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:05.379469 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.379534 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.379646 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.379664 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:13:05.379899 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.377363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45005 (local address 127.16.79.2:40491)
0620 23:13:05.377528 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:05.377576 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:05.378039 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:05.378303 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:05.378307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:05.378316 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:05.379292 (+ 976us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:05.379292 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:05.379318 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:05.379342 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:05.379533 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:05.379538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:05.379540 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:13:05.379642 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:13:05.379700 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260620 23:13:05.380398 21641 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45005
I20260620 23:13:05.380483 21641 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:05.380585 21641 heartbeater.cc:511] Master 127.16.79.62:45005 requested a full tablet report, sending...
I20260620 23:13:05.380941 21322 ts_manager.cc:194] Registered new tserver with Master: d74e12f8a00d409e983d01305ba5fd1c (127.16.79.2:37697)
I20260620 23:13:05.384492 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 21513
I20260620 23:13:05.384594 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:05.384837 16700 external_mini_cluster.cc:1428] Setting key 6a09d1c1edfc1f1f3aa9f0cd7cfaa61e
I20260620 23:13:05.385823 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45005
--builtin_ntp_servers=127.16.79.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:13:05.467052 21644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:05.467239 21644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:05.467260 21644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:05.468350 21644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:05.468382 21644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:05.468408 21644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:05.469807 21644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:05.469977 21644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:05.470198 21644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:05.475142 21649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:05.475287 21652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:05.475368 21644 server_base.cc:1034] running on GCE node
W20260620 23:13:05.475288 21650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:05.475881 21644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:05.476140 21644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:05.477332 21644 hybrid_clock.cc:648] HybridClock initialized: now 1781997185477300 us; error 58 us; skew 500 ppm
I20260620 23:13:05.478634 21644 webserver.cc:460] Webserver started at http://127.16.79.3:46277/ using document root <none> and password file <none>
I20260620 23:13:05.478880 21644 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:05.478951 21644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:05.479086 21644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:05.480149 21644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "36f7f0938324439b8a9a5c8bf1988e39"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "c585aec64e3c6fce1a5d212371a38dc9"
server_key_iv: "f2eaaaf603121ca3fba0fb154f0bd9eb"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.480507 21644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "36f7f0938324439b8a9a5c8bf1988e39"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "c585aec64e3c6fce1a5d212371a38dc9"
server_key_iv: "f2eaaaf603121ca3fba0fb154f0bd9eb"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.481819 21644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:05.482484 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:05.482652 21644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:05.482746 21644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "36f7f0938324439b8a9a5c8bf1988e39"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "c585aec64e3c6fce1a5d212371a38dc9"
server_key_iv: "f2eaaaf603121ca3fba0fb154f0bd9eb"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.482842 21644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:05.486089 21644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:05.486337 21644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:05.486487 21644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:05.486689 21644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:05.486987 21644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:05.487041 21644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:05.487079 21644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:05.487107 21644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:05.492810 21644 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:44137
I20260620 23:13:05.492852 21771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:44137 every 8 connection(s)
I20260620 23:13:05.493137 21644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:13:05.495934 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.493776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:46833 (local address 127.16.79.62:45005)
0620 23:13:05.493819 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.493821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.494105 (+ 284us) server_negotiation.cc:407] Connection header received
0620 23:13:05.494582 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.494583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.494589 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.494653 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.495705 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.495706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.495706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:05.495711 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.495724 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:05.495728 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.495781 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.495851 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.495866 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:13:05.496104 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.493820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45005 (local address 127.16.79.3:46833)
0620 23:13:05.493995 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:05.494024 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:05.494454 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:05.494717 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:05.494720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:05.494729 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:05.495564 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:05.495565 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:05.495588 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:05.495612 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:05.495778 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:05.495782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:05.495786 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:05.495849 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:13:05.495905 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260620 23:13:05.496675 21772 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45005
I20260620 23:13:05.497046 21772 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:05.497246 21772 heartbeater.cc:511] Master 127.16.79.62:45005 requested a full tablet report, sending...
I20260620 23:13:05.497597 21322 ts_manager.cc:194] Registered new tserver with Master: 36f7f0938324439b8a9a5c8bf1988e39 (127.16.79.3:44137)
I20260620 23:13:05.500120 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 21644
I20260620 23:13:05.500247 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:05.500404 16700 external_mini_cluster.cc:1428] Setting key efaf84ec641645e430770b095b89a7e3
I20260620 23:13:05.501205 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:13:05.503429 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.502422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.16.79.62:45005)
0620 23:13:05.502464 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.502466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.502553 (+ 87us) server_negotiation.cc:407] Connection header received
0620 23:13:05.502617 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.502617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.502621 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.502657 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.503217 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.503217 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.503218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:05.503221 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.503233 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:05.503237 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.503298 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.503325 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.503336 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:13:05.505030 21322 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:13:05.506508 21322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:13:05.510176 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.508730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.16.79.2:37697)
0620 23:13:05.508935 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.508940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.509194 (+ 254us) server_negotiation.cc:407] Connection header received
0620 23:13:05.509230 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.509232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.509242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.509503 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.509836 (+ 333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.509840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.509841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:05.509844 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.509874 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:05.509888 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.509983 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.510076 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.510092 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":96,"threads_started":1}
I20260620 23:13:05.510159 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.508504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:37697 (local address 127.0.0.1:60610)
0620 23:13:05.508683 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:05.509135 (+ 452us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:05.509186 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:05.509347 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:05.509348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:05.509351 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:05.509731 (+ 380us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:05.509732 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:05.509748 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:05.509771 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:05.509979 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:05.509984 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:05.509986 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:13:05.510064 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 23:13:05.510078 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":44,"threads_started":1}
I20260620 23:13:05.510241 21782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.508534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:44137 (local address 127.0.0.1:50524)
0620 23:13:05.509070 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:05.509077 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:05.509099 (+ 22us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:05.509431 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:05.509432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:05.509434 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:05.509871 (+ 437us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:05.509871 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:05.509877 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:05.509888 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:05.510094 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:05.510094 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:05.510095 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:13:05.510123 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 23:13:05.510135 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":68,"threads_started":1}
I20260620 23:13:05.510469 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.508608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.16.79.3:44137)
0620 23:13:05.509296 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.509300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.509309 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:05.509329 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.509330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.509337 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.509554 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.509968 (+ 414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.509973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.509974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:05.509977 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.510004 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:05.510019 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.510333 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.510361 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.510377 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":115,"threads_started":1}
I20260620 23:13:05.510558 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.508519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:37333 (local address 127.0.0.1:49286)
0620 23:13:05.508916 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:05.508955 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:05.509006 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:05.509149 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:05.509155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:05.509161 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:05.510175 (+ 1014us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:05.510175 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:05.510181 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:05.510194 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:05.510438 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:05.510439 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:05.510440 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:13:05.510487 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 23:13:05.510500 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":91,"threads_started":1}
I20260620 23:13:05.510587 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.508759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.16.79.1:37333)
0620 23:13:05.508939 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.508943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.509000 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:13:05.509046 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.509048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.509055 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.509146 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.510275 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.510280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.510281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:05.510284 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.510329 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:05.510354 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.510412 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.510496 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.510510 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260620 23:13:05.511135 21706 tablet_service.cc:1467] Processing CreateTablet for tablet d633b015b0c241a3a78f1926e372be59 (DEFAULT_TABLE table=test-table [id=5c6bde6f6edd44b2900398d0389d77eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:05.511317 21426 tablet_service.cc:1467] Processing CreateTablet for tablet d633b015b0c241a3a78f1926e372be59 (DEFAULT_TABLE table=test-table [id=5c6bde6f6edd44b2900398d0389d77eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:05.511358 21575 tablet_service.cc:1467] Processing CreateTablet for tablet d633b015b0c241a3a78f1926e372be59 (DEFAULT_TABLE table=test-table [id=5c6bde6f6edd44b2900398d0389d77eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:05.511476 21706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d633b015b0c241a3a78f1926e372be59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:05.511592 21426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d633b015b0c241a3a78f1926e372be59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:05.511669 21575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d633b015b0c241a3a78f1926e372be59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:05.514065 21786 tablet_bootstrap.cc:492] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c: Bootstrap starting.
I20260620 23:13:05.514698 21786 tablet_bootstrap.cc:654] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:05.514986 21786 log.cc:824] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:05.515739 21786 tablet_bootstrap.cc:492] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c: No bootstrap required, opened a new log
I20260620 23:13:05.515846 21786 ts_tablet_manager.cc:1397] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:13:05.515980 21788 tablet_bootstrap.cc:492] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39: Bootstrap starting.
I20260620 23:13:05.516180 21789 tablet_bootstrap.cc:492] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08: Bootstrap starting.
I20260620 23:13:05.516639 21788 tablet_bootstrap.cc:654] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:05.516939 21789 tablet_bootstrap.cc:654] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:05.516975 21788 log.cc:824] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:05.517452 21789 log.cc:824] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:05.517721 21788 tablet_bootstrap.cc:492] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39: No bootstrap required, opened a new log
I20260620 23:13:05.517789 21788 ts_tablet_manager.cc:1397] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:13:05.518112 21789 tablet_bootstrap.cc:492] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08: No bootstrap required, opened a new log
I20260620 23:13:05.518199 21789 ts_tablet_manager.cc:1397] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:13:05.518620 21786 raft_consensus.cc:348] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e12f8a00d409e983d01305ba5fd1c" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37697 } } peers { permanent_uuid: "1a24dfb4057b4f48b71b21bbcb075b08" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 37333 } } peers { permanent_uuid: "36f7f0938324439b8a9a5c8bf1988e39" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44137 } }
I20260620 23:13:05.518797 21786 raft_consensus.cc:374] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:05.518834 21786 raft_consensus.cc:729] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74e12f8a00d409e983d01305ba5fd1c, State: Initialized, Role: FOLLOWER
I20260620 23:13:05.518978 21786 consensus_queue.cc:260] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e12f8a00d409e983d01305ba5fd1c" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37697 } } peers { permanent_uuid: "1a24dfb4057b4f48b71b21bbcb075b08" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 37333 } } peers { permanent_uuid: "36f7f0938324439b8a9a5c8bf1988e39" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44137 } }
I20260620 23:13:05.519228 21786 ts_tablet_manager.cc:1428] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:13:05.519340 21641 heartbeater.cc:503] Master 127.16.79.62:45005 was elected leader, sending a full tablet report...
I20260620 23:13:05.520426 21788 raft_consensus.cc:348] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e12f8a00d409e983d01305ba5fd1c" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37697 } } peers { permanent_uuid: "1a24dfb4057b4f48b71b21bbcb075b08" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 37333 } } peers { permanent_uuid: "36f7f0938324439b8a9a5c8bf1988e39" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44137 } }
I20260620 23:13:05.520536 21788 raft_consensus.cc:374] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:05.520556 21788 raft_consensus.cc:729] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f7f0938324439b8a9a5c8bf1988e39, State: Initialized, Role: FOLLOWER
I20260620 23:13:05.520512 21789 raft_consensus.cc:348] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e12f8a00d409e983d01305ba5fd1c" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37697 } } peers { permanent_uuid: "1a24dfb4057b4f48b71b21bbcb075b08" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 37333 } } peers { permanent_uuid: "36f7f0938324439b8a9a5c8bf1988e39" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44137 } }
I20260620 23:13:05.520682 21789 raft_consensus.cc:374] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:05.520653 21788 consensus_queue.cc:260] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e12f8a00d409e983d01305ba5fd1c" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37697 } } peers { permanent_uuid: "1a24dfb4057b4f48b71b21bbcb075b08" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 37333 } } peers { permanent_uuid: "36f7f0938324439b8a9a5c8bf1988e39" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44137 } }
I20260620 23:13:05.520753 21789 raft_consensus.cc:729] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a24dfb4057b4f48b71b21bbcb075b08, State: Initialized, Role: FOLLOWER
I20260620 23:13:05.520915 21788 ts_tablet_manager.cc:1428] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:13:05.520920 21789 consensus_queue.cc:260] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e12f8a00d409e983d01305ba5fd1c" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37697 } } peers { permanent_uuid: "1a24dfb4057b4f48b71b21bbcb075b08" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 37333 } } peers { permanent_uuid: "36f7f0938324439b8a9a5c8bf1988e39" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44137 } }
I20260620 23:13:05.521008 21772 heartbeater.cc:503] Master 127.16.79.62:45005 was elected leader, sending a full tablet report...
I20260620 23:13:05.521179 21789 ts_tablet_manager.cc:1428] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:13:05.521214 21510 heartbeater.cc:503] Master 127.16.79.62:45005 was elected leader, sending a full tablet report...
I20260620 23:13:05.618260 21793 raft_consensus.cc:482] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:05.618391 21793 raft_consensus.cc:504] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e12f8a00d409e983d01305ba5fd1c" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37697 } } peers { permanent_uuid: "1a24dfb4057b4f48b71b21bbcb075b08" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 37333 } } peers { permanent_uuid: "36f7f0938324439b8a9a5c8bf1988e39" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44137 } }
I20260620 23:13:05.618827 21793 leader_election.cc:288] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d74e12f8a00d409e983d01305ba5fd1c (127.16.79.2:37697), 1a24dfb4057b4f48b71b21bbcb075b08 (127.16.79.1:37333)
I20260620 23:13:05.620328 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.619114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:40441 (local address 127.16.79.1:37333)
0620 23:13:05.619172 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.619174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.619311 (+ 137us) server_negotiation.cc:407] Connection header received
0620 23:13:05.619339 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.619340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.619345 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.619406 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.620060 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.620061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.620062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:05.620066 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.620083 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:05.620088 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.620155 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.620210 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.620224 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:13:05.620352 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.619056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:38123 (local address 127.16.79.2:37697)
0620 23:13:05.619099 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.619101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.619110 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:05.619192 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.619193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.619201 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.619275 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.620081 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.620082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.620083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:05.620087 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.620104 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:05.620108 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.620178 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.620253 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.620268 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:13:05.620344 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.619007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:37697 (local address 127.16.79.3:38123)
0620 23:13:05.619059 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:05.619071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:05.619111 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:05.619298 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:05.619299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:05.619302 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:05.619805 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:05.619806 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:05.619816 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:05.619837 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:05.620176 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:05.620177 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:05.620178 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:13:05.620248 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 23:13:05.620269 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 23:13:05.620357 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.619045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:37333 (local address 127.16.79.3:40441)
0620 23:13:05.619231 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:05.619241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:05.619290 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:05.619421 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:05.619421 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:05.619432 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:05.619952 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:05.619952 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:05.619961 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:05.619973 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:05.620165 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:05.620166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:05.620167 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:13:05.620213 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 23:13:05.620249 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":109,"threads_started":1}
I20260620 23:13:05.620682 21595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d633b015b0c241a3a78f1926e372be59" candidate_uuid: "36f7f0938324439b8a9a5c8bf1988e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74e12f8a00d409e983d01305ba5fd1c" is_pre_election: true
I20260620 23:13:05.620846 21595 raft_consensus.cc:2454] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f7f0938324439b8a9a5c8bf1988e39 in term 0.
I20260620 23:13:05.621029 21659 leader_election.cc:302] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f7f0938324439b8a9a5c8bf1988e39, d74e12f8a00d409e983d01305ba5fd1c; no voters:
I20260620 23:13:05.621088 21464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d633b015b0c241a3a78f1926e372be59" candidate_uuid: "36f7f0938324439b8a9a5c8bf1988e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a24dfb4057b4f48b71b21bbcb075b08" is_pre_election: true
I20260620 23:13:05.621179 21793 raft_consensus.cc:2781] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:13:05.621239 21793 raft_consensus.cc:482] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:05.621229 21464 raft_consensus.cc:2454] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f7f0938324439b8a9a5c8bf1988e39 in term 0.
I20260620 23:13:05.621270 21793 raft_consensus.cc:3037] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:05.622045 21793 raft_consensus.cc:504] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e12f8a00d409e983d01305ba5fd1c" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37697 } } peers { permanent_uuid: "1a24dfb4057b4f48b71b21bbcb075b08" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 37333 } } peers { permanent_uuid: "36f7f0938324439b8a9a5c8bf1988e39" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44137 } }
I20260620 23:13:05.622182 21793 leader_election.cc:288] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [CANDIDATE]: Term 1 election: Requested vote from peers d74e12f8a00d409e983d01305ba5fd1c (127.16.79.2:37697), 1a24dfb4057b4f48b71b21bbcb075b08 (127.16.79.1:37333)
I20260620 23:13:05.622323 21464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d633b015b0c241a3a78f1926e372be59" candidate_uuid: "36f7f0938324439b8a9a5c8bf1988e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a24dfb4057b4f48b71b21bbcb075b08"
I20260620 23:13:05.622334 21595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d633b015b0c241a3a78f1926e372be59" candidate_uuid: "36f7f0938324439b8a9a5c8bf1988e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74e12f8a00d409e983d01305ba5fd1c"
I20260620 23:13:05.622397 21595 raft_consensus.cc:3037] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:05.622397 21464 raft_consensus.cc:3037] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:05.623229 21595 raft_consensus.cc:2454] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f7f0938324439b8a9a5c8bf1988e39 in term 1.
I20260620 23:13:05.623229 21464 raft_consensus.cc:2454] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f7f0938324439b8a9a5c8bf1988e39 in term 1.
I20260620 23:13:05.623443 21660 leader_election.cc:302] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a24dfb4057b4f48b71b21bbcb075b08, 36f7f0938324439b8a9a5c8bf1988e39; no voters:
I20260620 23:13:05.623538 21793 raft_consensus.cc:2781] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:05.623678 21793 raft_consensus.cc:686] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 1 LEADER]: Becoming Leader. State: Replica: 36f7f0938324439b8a9a5c8bf1988e39, State: Running, Role: LEADER
I20260620 23:13:05.623785 21793 consensus_queue.cc:237] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e12f8a00d409e983d01305ba5fd1c" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37697 } } peers { permanent_uuid: "1a24dfb4057b4f48b71b21bbcb075b08" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 37333 } } peers { permanent_uuid: "36f7f0938324439b8a9a5c8bf1988e39" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44137 } }
I20260620 23:13:05.624526 21322 catalog_manager.cc:5495] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36f7f0938324439b8a9a5c8bf1988e39 (127.16.79.3). New cstate: current_term: 1 leader_uuid: "36f7f0938324439b8a9a5c8bf1988e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74e12f8a00d409e983d01305ba5fd1c" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a24dfb4057b4f48b71b21bbcb075b08" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36f7f0938324439b8a9a5c8bf1988e39" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44137 } health_report { overall_health: HEALTHY } } }
W20260620 23:13:05.627225 21642 tablet.cc:2369] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:13:05.630961 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.629775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.16.79.3:44137)
0620 23:13:05.629821 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.629823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.629832 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:05.629881 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.629882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.629898 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.629951 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.630681 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.630682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.630683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:05.630686 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.630702 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:05.630722 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.630782 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.630842 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.630855 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:13:05.632875 21595 raft_consensus.cc:1261] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c [term 1 FOLLOWER]: Refusing update from remote peer 36f7f0938324439b8a9a5c8bf1988e39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:05.632902 21464 raft_consensus.cc:1261] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08 [term 1 FOLLOWER]: Refusing update from remote peer 36f7f0938324439b8a9a5c8bf1988e39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:05.633307 21793 consensus_queue.cc:1035] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74e12f8a00d409e983d01305ba5fd1c" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:05.633466 21796 consensus_queue.cc:1035] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a24dfb4057b4f48b71b21bbcb075b08" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:05.636008 21799 mvcc.cc:205] Tried to move back new op lower bound from 7299060472349696000 to 7299060472315596800. Current Snapshot: MvccSnapshot[applied={T|T < 7299060472349696000}]
I20260620 23:13:05.638294 21322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:05.638387 21322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:05.639032 21322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed301cbc00cf48b1b526a4cf24f43707: Sending DeleteTablet for 3 replicas of tablet d633b015b0c241a3a78f1926e372be59
I20260620 23:13:05.639315 21575 tablet_service.cc:1514] Processing DeleteTablet for tablet d633b015b0c241a3a78f1926e372be59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:05 UTC) from {username='slave'} at 127.0.0.1:60610
I20260620 23:13:05.639420 21426 tablet_service.cc:1514] Processing DeleteTablet for tablet d633b015b0c241a3a78f1926e372be59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:05 UTC) from {username='slave'} at 127.0.0.1:49286
I20260620 23:13:05.639447 21706 tablet_service.cc:1514] Processing DeleteTablet for tablet d633b015b0c241a3a78f1926e372be59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:05 UTC) from {username='slave'} at 127.0.0.1:50524
I20260620 23:13:05.639617 21809 tablet_replica.cc:330] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c: stopping tablet replica
I20260620 23:13:05.639642 21811 tablet_replica.cc:330] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39: stopping tablet replica
I20260620 23:13:05.639703 21809 raft_consensus.cc:2229] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:05.639724 21811 raft_consensus.cc:2229] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:13:05.639788 21809 raft_consensus.cc:2258] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:05.639822 21811 raft_consensus.cc:2258] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:05.639925 21810 tablet_replica.cc:330] T d633b015b0c241a3a78f1926e372be59 P 1a24dfb4057b4f48b71b21bbcb075b08: stopping tablet replica
I20260620 23:13:05.639961 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 21382
I20260620 23:13:05.641717 21811 ts_tablet_manager.cc:1905] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:05.643297 21811 ts_tablet_manager.cc:1918] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:05.643389 21811 log.cc:1196] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/d633b015b0c241a3a78f1926e372be59
I20260620 23:13:05.643631 21811 ts_tablet_manager.cc:1939] T d633b015b0c241a3a78f1926e372be59 P 36f7f0938324439b8a9a5c8bf1988e39: Deleting consensus metadata
I20260620 23:13:05.643996 21308 catalog_manager.cc:4841] TS 36f7f0938324439b8a9a5c8bf1988e39 (127.16.79.3:44137): tablet d633b015b0c241a3a78f1926e372be59 (table test-table [id=5c6bde6f6edd44b2900398d0389d77eb]) successfully deleted
W20260620 23:13:05.648370 21307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:37333 (error 108)
I20260620 23:13:05.648823 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.648684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:37333 (local address 127.0.0.1:49294)
0620 23:13:05.648756 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:05.648792 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 23:13:05.648842 21809 ts_tablet_manager.cc:1905] T d633b015b0c241a3a78f1926e372be59 P d74e12f8a00d409e983d01305ba5fd1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:05.648898 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 21513
W20260620 23:13:05.652818 21307 catalog_manager.cc:4584] TS 1a24dfb4057b4f48b71b21bbcb075b08 (127.16.79.1:37333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d633b015b0c241a3a78f1926e372be59: Network error: Client connection negotiation failed: client connection to 127.16.79.1:37333: connect: Connection refused (error 111)
I20260620 23:13:05.653347 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 21644
I20260620 23:13:05.653381 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.653280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:37697 (local address 127.0.0.1:60624)
0620 23:13:05.653326 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:05.653354 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.2:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 23:13:05.657908 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 21291
2026-06-20T23:13:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (670 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 23:13:05.677215 16700 mini_oidc.cc:152] Starting JWKS server
I20260620 23:13:05.678179 16700 webserver.cc:460] Webserver started at https://127.0.0.1:40217/ using document root <none> and password file <none>
I20260620 23:13:05.678301 16700 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:13:05.678472 16700 webserver.cc:460] Webserver started at http://127.0.0.1:43827/ using document root <none> and password file <none>
2026-06-20T23:13:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:05Z Disabled control of system clock
I20260620 23:13:05.685813 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36599
--webserver_interface=127.16.79.62
--webserver_port=0
--builtin_ntp_servers=127.16.79.20:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:36599
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40217/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/testchainca.cert with env {}
W20260620 23:13:05.763088 21818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:05.763271 21818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:05.763293 21818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:05.764338 21818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:05.764372 21818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:05.764384 21818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:05.764397 21818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:05.764410 21818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:13:05.764421 21818 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:13:05.764432 21818 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40217/jwks.json
I20260620 23:13:05.765875 21818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:36599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40217/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:05.766052 21818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:05.766204 21818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:05.770880 21823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:05.770910 21826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:05.770988 21824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:05.771740 21818 server_base.cc:1034] running on GCE node
I20260620 23:13:05.771988 21818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:05.772248 21818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:05.773406 21818 hybrid_clock.cc:648] HybridClock initialized: now 1781997185773388 us; error 40 us; skew 500 ppm
I20260620 23:13:05.774658 21818 webserver.cc:460] Webserver started at http://127.16.79.62:43059/ using document root <none> and password file <none>
I20260620 23:13:05.774899 21818 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:05.774966 21818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:05.775105 21818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:05.776078 21818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "32c32d6dc2f542a79fe2b8eb298902b8"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "b97e470b5b506f8e45797a284d4d4cc6"
server_key_iv: "4a979f527a2bd29d020f27d50fabc74f"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.776412 21818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "32c32d6dc2f542a79fe2b8eb298902b8"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "b97e470b5b506f8e45797a284d4d4cc6"
server_key_iv: "4a979f527a2bd29d020f27d50fabc74f"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.777770 21818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:13:05.778491 21832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:05.778659 21818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:05.778749 21818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "32c32d6dc2f542a79fe2b8eb298902b8"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-pswb"
server_key: "b97e470b5b506f8e45797a284d4d4cc6"
server_key_iv: "4a979f527a2bd29d020f27d50fabc74f"
server_key_version: "encryptionkey@0"
I20260620 23:13:05.778827 21818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:05.791363 21818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:05.791625 21818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:05.791766 21818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:05.795629 21818 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:36599
I20260620 23:13:05.795662 21884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:36599 every 8 connection(s)
I20260620 23:13:05.795962 21818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:05.796962 21885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:05.799094 21885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8: Bootstrap starting.
I20260620 23:13:05.799664 21885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:05.799754 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 21818
I20260620 23:13:05.799901 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:05.800041 21885 log.cc:824] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:05.800077 16700 external_mini_cluster.cc:1428] Setting key 93546d21717a45a46f535002676766ec
I20260620 23:13:05.800876 21885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8: No bootstrap required, opened a new log
I20260620 23:13:05.803172 21885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c32d6dc2f542a79fe2b8eb298902b8" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36599 } }
I20260620 23:13:05.803329 21885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:05.803360 21885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c32d6dc2f542a79fe2b8eb298902b8, State: Initialized, Role: FOLLOWER
I20260620 23:13:05.803469 21885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c32d6dc2f542a79fe2b8eb298902b8" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36599 } }
I20260620 23:13:05.803548 21885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:05.803582 21885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:05.803663 21885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:05.804234 21885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c32d6dc2f542a79fe2b8eb298902b8" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36599 } }
I20260620 23:13:05.804350 21885 leader_election.cc:302] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32c32d6dc2f542a79fe2b8eb298902b8; no voters:
I20260620 23:13:05.804535 21885 leader_election.cc:288] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:05.804560 21890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:05.804670 21890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [term 1 LEADER]: Becoming Leader. State: Replica: 32c32d6dc2f542a79fe2b8eb298902b8, State: Running, Role: LEADER
I20260620 23:13:05.804764 21890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c32d6dc2f542a79fe2b8eb298902b8" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36599 } }
I20260620 23:13:05.804837 21885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:05.805127 21890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32c32d6dc2f542a79fe2b8eb298902b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c32d6dc2f542a79fe2b8eb298902b8" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36599 } } }
I20260620 23:13:05.805279 21890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:05.805475 21890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c32d6dc2f542a79fe2b8eb298902b8. Latest consensus state: current_term: 1 leader_uuid: "32c32d6dc2f542a79fe2b8eb298902b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c32d6dc2f542a79fe2b8eb298902b8" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36599 } } }
I20260620 23:13:05.805598 21890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:05.805790 21889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.800565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.16.79.62:36599)
0620 23:13:05.800781 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.800787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.800810 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:13:05.801245 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.801251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.801305 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.801408 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.801623 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:05.802429 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:05.803577 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:05.804059 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:05.804158 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:05.804888 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.804893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.804893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:05.805112 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.805153 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:05.805167 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.805225 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.805484 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.805562 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260620 23:13:05.805650 21897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:05.806519 21897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:05.806689 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:13:05.807902 21897 catalog_manager.cc:1347] Generated new cluster ID: cdd3abbdd580436aa374956e4af3c1e0
I20260620 23:13:05.807952 21897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:05.837508 21897 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:05.838035 21897 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:05.842170 21897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32c32d6dc2f542a79fe2b8eb298902b8: Generated new TSK 0
I20260620 23:13:05.842338 21897 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:13:05.871205 21915 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 23:13:05.871305 21915 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:13:05.871541 21889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.855814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.16.79.62:36599)
0620 23:13:05.855865 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.855867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.855977 (+ 110us) server_negotiation.cc:407] Connection header received
0620 23:13:05.856044 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.856045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.856056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.856131 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:13:05.856225 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:05.856948 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:05.858221 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:05.858526 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:05.858600 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:05.858823 (+ 223us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:13:05.871328 (+ 12505us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:13:05.871471 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60872: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 23:13:05.871601 21889 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60872: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 23:13:05.871978 21914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:05.855737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:36599 (local address 127.0.0.1:60872)
0620 23:13:05.855914 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:05.855923 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:05.855980 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:05.856131 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:05.856132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:05.856133 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:13:05.856182 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:05.856184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:05.857051 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:05.857052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:05.858130 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:05.858133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:05.858636 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:05.858637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:05.858686 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:05.858704 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:13:05.871483 (+ 12779us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:13:05.871576 (+ 93us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.79.62:36599: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.79.62:36599: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 23:13:05.872452 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 21818
2026-06-20T23:13:05Z chronyd exiting
I20260620 23:13:05.908595 16700 test_util.cc:175] -----------------------------------------------
I20260620 23:13:05.908661 16700 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997163808439-16700-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 23:13:05.909461 16700 mini_oidc.cc:152] Starting JWKS server
I20260620 23:13:05.910116 16700 webserver.cc:460] Webserver started at https://127.0.0.1:37873/ using document root <none> and password file <none>
I20260620 23:13:05.910156 16700 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:13:05.910337 16700 webserver.cc:460] Webserver started at http://127.0.0.1:33553/ using document root <none> and password file <none>
2026-06-20T23:13:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:05Z Disabled control of system clock
I20260620 23:13:05.918718 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:35611
--webserver_interface=127.16.79.62
--webserver_port=0
--builtin_ntp_servers=127.16.79.20:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:35611
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37873/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/testchainca.cert with env {}
W20260620 23:13:05.996062 21921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:05.996294 21921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:05.996330 21921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:05.997524 21921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:05.997588 21921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:05.997611 21921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:05.997639 21921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:05.997671 21921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:13:05.997694 21921 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:13:05.997715 21921 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37873/jwks.json
I20260620 23:13:05.999158 21921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:35611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:35611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37873/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:05.999425 21921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:05.999629 21921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:06.005123 21927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:06.005141 21929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:06.005187 21926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:06.005499 21921 server_base.cc:1034] running on GCE node
I20260620 23:13:06.005781 21921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:06.006062 21921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:06.007212 21921 hybrid_clock.cc:648] HybridClock initialized: now 1781997186007199 us; error 26 us; skew 500 ppm
I20260620 23:13:06.008416 21921 webserver.cc:460] Webserver started at http://127.16.79.62:45899/ using document root <none> and password file <none>
I20260620 23:13:06.008641 21921 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:06.008687 21921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:06.008865 21921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:06.009924 21921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "5e89fcb3cdf2404f97e3e1deef511404"
format_stamp: "Formatted at 2026-06-20 23:13:06 on dist-test-slave-pswb"
server_key: "b92a24678f16d01f489c86d2a969dd2c"
server_key_iv: "158a39b4cc1ba02b49a8100286b008ba"
server_key_version: "encryptionkey@0"
I20260620 23:13:06.010278 21921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "5e89fcb3cdf2404f97e3e1deef511404"
format_stamp: "Formatted at 2026-06-20 23:13:06 on dist-test-slave-pswb"
server_key: "b92a24678f16d01f489c86d2a969dd2c"
server_key_iv: "158a39b4cc1ba02b49a8100286b008ba"
server_key_version: "encryptionkey@0"
I20260620 23:13:06.011744 21921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:13:06.012591 21935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:06.012852 21921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:06.012957 21921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "5e89fcb3cdf2404f97e3e1deef511404"
format_stamp: "Formatted at 2026-06-20 23:13:06 on dist-test-slave-pswb"
server_key: "b92a24678f16d01f489c86d2a969dd2c"
server_key_iv: "158a39b4cc1ba02b49a8100286b008ba"
server_key_version: "encryptionkey@0"
I20260620 23:13:06.013047 21921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:06.036254 21921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:06.036556 21921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:06.036751 21921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:06.040792 21921 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:35611
I20260620 23:13:06.040825 21987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:35611 every 8 connection(s)
I20260620 23:13:06.041172 21921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:06.041813 21988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:06.042909 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 21921
I20260620 23:13:06.043012 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:06.043188 16700 external_mini_cluster.cc:1428] Setting key 93000e4da53cfa3562b6acf88343f706
I20260620 23:13:06.045186 21988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404: Bootstrap starting.
I20260620 23:13:06.045857 21988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:06.046190 21988 log.cc:824] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:06.046924 21988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404: No bootstrap required, opened a new log
I20260620 23:13:06.048825 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:06.043642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.16.79.62:35611)
0620 23:13:06.043803 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:06.043819 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:06.043843 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:13:06.044263 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:06.044268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:06.044342 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:06.044450 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:06.044729 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.045631 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.046769 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.047186 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.047274 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:06.048136 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:06.048139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:06.048140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:06.048287 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:06.048334 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:06.048344 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:06.048396 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:06.048495 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 23:13:06.048547 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
I20260620 23:13:06.049400 21988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89fcb3cdf2404f97e3e1deef511404" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35611 } }
I20260620 23:13:06.049553 21988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:06.049604 21988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e89fcb3cdf2404f97e3e1deef511404, State: Initialized, Role: FOLLOWER
I20260620 23:13:06.049630 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:13:06.049731 21988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89fcb3cdf2404f97e3e1deef511404" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35611 } }
I20260620 23:13:06.049826 21988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:06.049875 21988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:06.049926 21988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:06.050462 21988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89fcb3cdf2404f97e3e1deef511404" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35611 } }
I20260620 23:13:06.050578 21988 leader_election.cc:302] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e89fcb3cdf2404f97e3e1deef511404; no voters:
I20260620 23:13:06.050738 21988 leader_election.cc:288] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:06.050794 21993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:06.050953 21993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [term 1 LEADER]: Becoming Leader. State: Replica: 5e89fcb3cdf2404f97e3e1deef511404, State: Running, Role: LEADER
I20260620 23:13:06.051105 21993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89fcb3cdf2404f97e3e1deef511404" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35611 } }
I20260620 23:13:06.051182 21988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:06.051607 21994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e89fcb3cdf2404f97e3e1deef511404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89fcb3cdf2404f97e3e1deef511404" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35611 } } }
I20260620 23:13:06.051697 21994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:06.051940 21997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:06.051957 21995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e89fcb3cdf2404f97e3e1deef511404. Latest consensus state: current_term: 1 leader_uuid: "5e89fcb3cdf2404f97e3e1deef511404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89fcb3cdf2404f97e3e1deef511404" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35611 } } }
I20260620 23:13:06.052146 21995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:06.052526 21997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:06.054010 21997 catalog_manager.cc:1347] Generated new cluster ID: 76d75c67be2a4fc9b57d018aa0aac5d0
I20260620 23:13:06.054064 21997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:06.063293 21997 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:06.063791 21997 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:06.070309 21997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e89fcb3cdf2404f97e3e1deef511404: Generated new TSK 0
I20260620 23:13:06.070502 21997 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:13:06.107893 22018 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 23:13:06.107970 22018 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:13:06.108247 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:06.085904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.16.79.62:35611)
0620 23:13:06.085942 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:06.085946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:06.085977 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:13:06.086001 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:06.086002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:06.086012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:06.086064 (+ 52us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:13:06.086223 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.086725 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.088026 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.088270 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.088362 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:06.088584 (+ 222us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:13:06.108044 (+ 19460us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:13:06.108196 (+ 152us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 23:13:06.108325 21991 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 23:13:06.108358 22017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:06.085721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:35611 (local address 127.0.0.1:52866)
0620 23:13:06.085873 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:06.085883 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:06.085929 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:06.086085 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:06.086086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:06.086088 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:13:06.086155 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:06.086157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.086819 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.086820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:06.087772 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:06.087774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.088380 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.088381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:06.088449 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:06.088454 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:13:06.108162 (+ 19708us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:13:06.108267 (+ 105us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.79.62:35611: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260620 23:13:06.108762 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 21921
2026-06-20T23:13:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 23:13:06.146522 16700 mini_oidc.cc:152] Starting JWKS server
I20260620 23:13:06.147197 16700 webserver.cc:460] Webserver started at https://127.0.0.1:43499/ using document root <none> and password file <none>
I20260620 23:13:06.147265 16700 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:13:06.147432 16700 webserver.cc:460] Webserver started at http://127.0.0.1:36857/ using document root <none> and password file <none>
2026-06-20T23:13:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:06Z Disabled control of system clock
I20260620 23:13:06.154614 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:33463
--webserver_interface=127.16.79.62
--webserver_port=0
--builtin_ntp_servers=127.16.79.20:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:33463
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43499/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:13:06.243044 22024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:06.243261 22024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:06.243305 22024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:06.244415 22024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:06.244480 22024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:06.244503 22024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:06.244531 22024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:06.244557 22024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:13:06.244586 22024 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:13:06.244632 22024 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43499/jwks.json
I20260620 23:13:06.245970 22024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:33463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:33463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43499/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:06.246243 22024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:06.246510 22024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:06.251502 22032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:06.251581 22029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:06.251751 22024 server_base.cc:1034] running on GCE node
W20260620 23:13:06.251938 22030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:06.252267 22024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:06.252662 22024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:06.253912 22024 hybrid_clock.cc:648] HybridClock initialized: now 1781997186253888 us; error 46 us; skew 500 ppm
I20260620 23:13:06.255184 22024 webserver.cc:460] Webserver started at http://127.16.79.62:43601/ using document root <none> and password file <none>
I20260620 23:13:06.255412 22024 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:06.255457 22024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:06.255620 22024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:06.256789 22024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "52b3de4bbe8e423b8598ea6a5aa9c163"
format_stamp: "Formatted at 2026-06-20 23:13:06 on dist-test-slave-pswb"
server_key: "a145f3838d10824b8c0a6ca01b2cce24"
server_key_iv: "708c7005523afacd899d1b0f1912b596"
server_key_version: "encryptionkey@0"
I20260620 23:13:06.257146 22024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "52b3de4bbe8e423b8598ea6a5aa9c163"
format_stamp: "Formatted at 2026-06-20 23:13:06 on dist-test-slave-pswb"
server_key: "a145f3838d10824b8c0a6ca01b2cce24"
server_key_iv: "708c7005523afacd899d1b0f1912b596"
server_key_version: "encryptionkey@0"
I20260620 23:13:06.258419 22024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:06.259326 22038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:06.259545 22024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:13:06.259614 22024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "52b3de4bbe8e423b8598ea6a5aa9c163"
format_stamp: "Formatted at 2026-06-20 23:13:06 on dist-test-slave-pswb"
server_key: "a145f3838d10824b8c0a6ca01b2cce24"
server_key_iv: "708c7005523afacd899d1b0f1912b596"
server_key_version: "encryptionkey@0"
I20260620 23:13:06.259701 22024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:06.271348 22024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:06.271572 22024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:06.271706 22024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:06.275501 22024 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:33463
I20260620 23:13:06.275561 22090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:33463 every 8 connection(s)
I20260620 23:13:06.275871 22024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:06.276935 22091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:06.279687 22091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163: Bootstrap starting.
I20260620 23:13:06.280253 22091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:06.280532 22091 log.cc:824] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:06.281330 22091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163: No bootstrap required, opened a new log
I20260620 23:13:06.282087 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 22024
I20260620 23:13:06.282187 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:06.282382 16700 external_mini_cluster.cc:1428] Setting key 8b6fd9a9a73aa861a620468a3106e40e
I20260620 23:13:06.283535 22091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b3de4bbe8e423b8598ea6a5aa9c163" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33463 } }
I20260620 23:13:06.283666 22091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:06.283687 22091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3de4bbe8e423b8598ea6a5aa9c163, State: Initialized, Role: FOLLOWER
I20260620 23:13:06.283831 22091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b3de4bbe8e423b8598ea6a5aa9c163" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33463 } }
I20260620 23:13:06.283926 22091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:06.283967 22091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:06.284039 22091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:06.284592 22091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b3de4bbe8e423b8598ea6a5aa9c163" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33463 } }
I20260620 23:13:06.284725 22091 leader_election.cc:302] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52b3de4bbe8e423b8598ea6a5aa9c163; no voters:
I20260620 23:13:06.284934 22091 leader_election.cc:288] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:06.284958 22096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:06.285065 22096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [term 1 LEADER]: Becoming Leader. State: Replica: 52b3de4bbe8e423b8598ea6a5aa9c163, State: Running, Role: LEADER
I20260620 23:13:06.285214 22091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:06.285190 22096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b3de4bbe8e423b8598ea6a5aa9c163" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33463 } }
I20260620 23:13:06.285621 22097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52b3de4bbe8e423b8598ea6a5aa9c163. Latest consensus state: current_term: 1 leader_uuid: "52b3de4bbe8e423b8598ea6a5aa9c163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b3de4bbe8e423b8598ea6a5aa9c163" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33463 } } }
I20260620 23:13:06.285707 22097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:06.285936 22098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52b3de4bbe8e423b8598ea6a5aa9c163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b3de4bbe8e423b8598ea6a5aa9c163" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33463 } } }
I20260620 23:13:06.286214 22098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:06.286432 22103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:06.286954 22103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:06.288756 22103 catalog_manager.cc:1347] Generated new cluster ID: f332da79a8e44a10b91f11a923b6821a
I20260620 23:13:06.288810 22103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:06.293275 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:06.283093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.16.79.62:33463)
0620 23:13:06.283415 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:06.283422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:06.283457 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:13:06.283557 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:06.283562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:06.283625 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:06.283800 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:06.284059 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.285511 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.286651 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.287309 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.287426 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:06.288125 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:06.288129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:06.288129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:06.288335 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:06.288374 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:06.288387 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:06.288458 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:06.292917 (+ 4459us) server_negotiation.cc:299] Negotiation successful
0620 23:13:06.293037 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":112,"threads_started":1}
I20260620 23:13:06.294778 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:13:06.317510 22103 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:06.318173 22103 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:06.333550 22103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52b3de4bbe8e423b8598ea6a5aa9c163: Generated new TSK 0
I20260620 23:13:06.333841 22103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:13:06.349964 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:06.346390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.16.79.62:33463)
0620 23:13:06.346470 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:06.346473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:06.346640 (+ 167us) server_negotiation.cc:407] Connection header received
0620 23:13:06.346693 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:06.346694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:06.346705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:06.346761 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:13:06.346939 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.347420 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.348331 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.348530 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.348626 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:06.348821 (+ 195us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:13:06.349757 (+ 936us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:13:06.349893 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42788: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 23:13:06.350024 22095 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42788: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 23:13:06.350085 22120 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:06.346307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33463 (local address 127.0.0.1:42788)
0620 23:13:06.346515 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:06.346530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:06.346620 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:06.346786 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:06.346787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:06.346789 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:13:06.346867 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:06.346870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.347541 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.347542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:06.348234 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:06.348236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.348640 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.348641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:06.348721 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:06.348726 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:13:06.349875 (+ 1149us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:13:06.349978 (+ 103us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.79.62:33463: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":120,"threads_started":1}
I20260620 23:13:06.350730 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 22024
E20260620 23:13:06.361191 22121 webserver.cc:533] Webserver: sslize failed:
E20260620 23:13:06.361250 22121 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T23:13:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (241 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T23:13:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:06Z Disabled control of system clock
I20260620 23:13:06.396162 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:38839
--webserver_interface=127.16.79.62
--webserver_port=0
--builtin_ntp_servers=127.16.79.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:38839
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/testchainca.cert with env {}
W20260620 23:13:06.476181 22125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:06.476389 22125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:06.476424 22125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:06.477558 22125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:06.477622 22125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:06.477644 22125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:06.477676 22125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:06.477703 22125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:13:06.477728 22125 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:13:06.477756 22125 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 23:13:06.479187 22125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:38839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:38839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:06.479435 22125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:06.479636 22125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:06.484359 22131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:06.484469 22130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:06.484375 22133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:06.485026 22125 server_base.cc:1034] running on GCE node
I20260620 23:13:06.485307 22125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:06.485616 22125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:06.486809 22125 hybrid_clock.cc:648] HybridClock initialized: now 1781997186486808 us; error 45 us; skew 500 ppm
I20260620 23:13:06.488113 22125 webserver.cc:460] Webserver started at http://127.16.79.62:38351/ using document root <none> and password file <none>
I20260620 23:13:06.488377 22125 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:06.488453 22125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:06.488612 22125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:06.489758 22125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "084ac9e0212f4233b8977ac9fb3b4ae2"
format_stamp: "Formatted at 2026-06-20 23:13:06 on dist-test-slave-pswb"
server_key: "4be260fcbe64430ac983c97cfbe6551e"
server_key_iv: "aa85fcc15dbef42731b3c76f5254e8da"
server_key_version: "encryptionkey@0"
I20260620 23:13:06.490132 22125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "084ac9e0212f4233b8977ac9fb3b4ae2"
format_stamp: "Formatted at 2026-06-20 23:13:06 on dist-test-slave-pswb"
server_key: "4be260fcbe64430ac983c97cfbe6551e"
server_key_iv: "aa85fcc15dbef42731b3c76f5254e8da"
server_key_version: "encryptionkey@0"
I20260620 23:13:06.491444 22125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:13:06.492264 22139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:06.492517 22125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:06.492623 22125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "084ac9e0212f4233b8977ac9fb3b4ae2"
format_stamp: "Formatted at 2026-06-20 23:13:06 on dist-test-slave-pswb"
server_key: "4be260fcbe64430ac983c97cfbe6551e"
server_key_iv: "aa85fcc15dbef42731b3c76f5254e8da"
server_key_version: "encryptionkey@0"
I20260620 23:13:06.492733 22125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:06.503906 22125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:06.504180 22125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:06.504329 22125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:06.508108 22125 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:38839
I20260620 23:13:06.508183 22191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:38839 every 8 connection(s)
I20260620 23:13:06.508509 22125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:06.509460 22192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:06.511065 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 22125
I20260620 23:13:06.511181 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:06.511353 16700 external_mini_cluster.cc:1428] Setting key 61c84ad6944e6920e3a9e356d1cc7f34
I20260620 23:13:06.511893 22192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2: Bootstrap starting.
I20260620 23:13:06.512564 22192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:06.512913 22192 log.cc:824] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:06.513564 22192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2: No bootstrap required, opened a new log
I20260620 23:13:06.515270 22192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084ac9e0212f4233b8977ac9fb3b4ae2" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 38839 } }
I20260620 23:13:06.515419 22192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:06.515467 22192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 084ac9e0212f4233b8977ac9fb3b4ae2, State: Initialized, Role: FOLLOWER
I20260620 23:13:06.515643 22192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084ac9e0212f4233b8977ac9fb3b4ae2" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 38839 } }
I20260620 23:13:06.515748 22192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:06.515779 22192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:06.515815 22192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:06.516438 22192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084ac9e0212f4233b8977ac9fb3b4ae2" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 38839 } }
I20260620 23:13:06.516541 22192 leader_election.cc:302] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 084ac9e0212f4233b8977ac9fb3b4ae2; no voters:
I20260620 23:13:06.516680 22192 leader_election.cc:288] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:06.516752 22197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:06.516985 22192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:06.517174 22197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [term 1 LEADER]: Becoming Leader. State: Replica: 084ac9e0212f4233b8977ac9fb3b4ae2, State: Running, Role: LEADER
I20260620 23:13:06.517293 22197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084ac9e0212f4233b8977ac9fb3b4ae2" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 38839 } }
I20260620 23:13:06.517704 22199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 084ac9e0212f4233b8977ac9fb3b4ae2. Latest consensus state: current_term: 1 leader_uuid: "084ac9e0212f4233b8977ac9fb3b4ae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084ac9e0212f4233b8977ac9fb3b4ae2" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 38839 } } }
I20260620 23:13:06.517782 22199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:06.517848 22198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "084ac9e0212f4233b8977ac9fb3b4ae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084ac9e0212f4233b8977ac9fb3b4ae2" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 38839 } } }
I20260620 23:13:06.517930 22198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:06.518050 22206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:06.518425 22195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:06.512105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.16.79.62:38839)
0620 23:13:06.512448 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:06.512455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:06.512481 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:13:06.512982 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:06.512988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:06.513085 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:06.513236 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:06.513593 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.514710 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.515621 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.516275 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.516424 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:06.517307 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:06.517311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:06.517311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:06.517507 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:06.517547 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:06.517580 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:06.517645 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:06.518121 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 23:13:06.518217 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":133,"threads_started":1}
I20260620 23:13:06.518658 22206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:06.519374 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:13:06.520149 22206 catalog_manager.cc:1347] Generated new cluster ID: 2831203ebab746d3a12052bab4e0cf1e
I20260620 23:13:06.520200 22206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:06.561316 22206 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:06.562041 22206 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:06.582276 22206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 084ac9e0212f4233b8977ac9fb3b4ae2: Generated new TSK 0
I20260620 23:13:06.582461 22206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:13:06.685114 22195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:06.598191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.16.79.62:38839)
0620 23:13:06.598244 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:06.598248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:06.598266 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:13:06.598374 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:06.598375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:06.598389 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:06.598492 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:13:06.598720 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.599270 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.600446 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.600640 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.600776 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:06.600964 (+ 188us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:13:06.684887 (+ 83923us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:13:06.685041 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35658: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 23:13:06.685197 22195 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35658: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 23:13:06.685316 22221 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:06.598023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:38839 (local address 127.0.0.1:35658)
0620 23:13:06.598201 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:06.598212 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:06.598268 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:06.598523 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:06.598525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:06.598527 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:13:06.598597 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:06.598600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.599402 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.599403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:06.600344 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:06.600347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.600795 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.600796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:06.600855 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:06.600861 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:13:06.685053 (+ 84192us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:13:06.685178 (+ 125us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.79.62:38839: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260620 23:13:06.685748 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 22125
2026-06-20T23:13:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (316 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:13:06 dist-test-slave-pswb krb5kdc[22233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:13:06 dist-test-slave-pswb krb5kdc[22233](info): set up 2 sockets
Jun 20 23:13:06 dist-test-slave-pswb krb5kdc[22233](info): commencing operation
krb5kdc: starting...
W20260620 23:13:07.928927 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:13:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:13:08.045682 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45027
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:45027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:08.122278 22246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:08.122485 22246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:08.122519 22246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:08.123623 22246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:08.123687 22246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:08.123713 22246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:08.123744 22246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:08.123767 22246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:08.125168 22246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:45027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:08.125417 22246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:08.125633 22246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:08.130416 22252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:08.130462 22251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:08.130523 22246 server_base.cc:1034] running on GCE node
W20260620 23:13:08.130462 22254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:08.131002 22246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:08.131337 22246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:08.132562 22246 hybrid_clock.cc:648] HybridClock initialized: now 1781997188132523 us; error 53 us; skew 500 ppm
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:08.135524 22246 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:13:08.135890 22246 webserver.cc:460] Webserver started at http://127.16.79.62:46525/ using document root <none> and password file <none>
I20260620 23:13:08.136070 22246 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:08.136109 22246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:08.136191 22246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:08.137192 22246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "18d6dd6783a64b88945824823d72b3f0"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-pswb"
server_key: "7361af7c8dbce79e7e9c730bfdd77da5"
server_key_iv: "35162bd2e2835918a825bd4ee8d0c349"
server_key_version: "encryptionkey@0"
I20260620 23:13:08.137475 22246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "18d6dd6783a64b88945824823d72b3f0"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-pswb"
server_key: "7361af7c8dbce79e7e9c730bfdd77da5"
server_key_iv: "35162bd2e2835918a825bd4ee8d0c349"
server_key_version: "encryptionkey@0"
I20260620 23:13:08.138789 22246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:08.139971 22261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:08.140190 22246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:08.140305 22246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "18d6dd6783a64b88945824823d72b3f0"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-pswb"
server_key: "7361af7c8dbce79e7e9c730bfdd77da5"
server_key_iv: "35162bd2e2835918a825bd4ee8d0c349"
server_key_version: "encryptionkey@0"
I20260620 23:13:08.140378 22246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:08.160041 22246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:08.162690 22246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:08.162871 22246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:08.167330 22246 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:45027
I20260620 23:13:08.167387 22313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:45027 every 8 connection(s)
I20260620 23:13:08.167763 22246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:08.168884 22314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:08.169924 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 22246
I20260620 23:13:08.170039 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:08.170210 16700 external_mini_cluster.cc:1428] Setting key 594b8556a796cdb454b65921d7fd578f
I20260620 23:13:08.172143 22314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0: Bootstrap starting.
I20260620 23:13:08.172827 22314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:08.173125 22314 log.cc:824] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:08.173822 22314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0: No bootstrap required, opened a new log
I20260620 23:13:08.176011 22314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d6dd6783a64b88945824823d72b3f0" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45027 } }
I20260620 23:13:08.176131 22314 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:08.176151 22314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d6dd6783a64b88945824823d72b3f0, State: Initialized, Role: FOLLOWER
I20260620 23:13:08.176287 22314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d6dd6783a64b88945824823d72b3f0" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45027 } }
I20260620 23:13:08.176359 22314 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:08.176380 22314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:08.176445 22314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:08.177011 22314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d6dd6783a64b88945824823d72b3f0" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45027 } }
I20260620 23:13:08.177126 22314 leader_election.cc:302] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18d6dd6783a64b88945824823d72b3f0; no voters:
I20260620 23:13:08.177309 22314 leader_election.cc:288] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:08.177345 22319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:08.177493 22319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [term 1 LEADER]: Becoming Leader. State: Replica: 18d6dd6783a64b88945824823d72b3f0, State: Running, Role: LEADER
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:08.177595 22314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:08.177623 22319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d6dd6783a64b88945824823d72b3f0" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45027 } }
I20260620 23:13:08.177933 22319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18d6dd6783a64b88945824823d72b3f0. Latest consensus state: current_term: 1 leader_uuid: "18d6dd6783a64b88945824823d72b3f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d6dd6783a64b88945824823d72b3f0" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45027 } } }
I20260620 23:13:08.177945 22320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18d6dd6783a64b88945824823d72b3f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d6dd6783a64b88945824823d72b3f0" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45027 } } }
I20260620 23:13:08.178148 22320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:08.178063 22319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:08.178416 22328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:08.178988 22328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:08.180835 22328 catalog_manager.cc:1347] Generated new cluster ID: 7f0ab749152644f69a23470a35ff3e97
I20260620 23:13:08.180899 22328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:08.181586 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.170671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.16.79.62:45027)
0620 23:13:08.170830 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.170839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.170868 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:13:08.171412 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.171421 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.171482 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.171631 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.172529 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.173931 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.174725 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.175338 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.175477 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.178112 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:08.178116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:08.178116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:08.178127 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:08.179966 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.180355 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.180358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.180359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.180459 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.180678 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.180679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.180680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.181008 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:08.181142 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.181272 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.181401 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260620 23:13:08.202495 22328 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:08.202977 22328 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:08.215417 22328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18d6dd6783a64b88945824823d72b3f0: Generated new TSK 0
I20260620 23:13:08.215798 22328 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:13:08.247699 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45027
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:08.345942 22340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:08.346133 22340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:08.346155 22340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:08.347254 22340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:08.347298 22340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:08.347324 22340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:08.348744 22340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:08.348953 22340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:08.349164 22340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:08.353919 22346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:08.353962 22348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:08.353962 22345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:08.354317 22340 server_base.cc:1034] running on GCE node
I20260620 23:13:08.354549 22340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:08.354802 22340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:08.355973 22340 hybrid_clock.cc:648] HybridClock initialized: now 1781997188355948 us; error 51 us; skew 500 ppm
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:08.358875 22340 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:13:08.359311 22340 webserver.cc:460] Webserver started at http://127.16.79.1:46805/ using document root <none> and password file <none>
I20260620 23:13:08.359542 22340 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:08.359608 22340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:08.359898 22340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:08.361263 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "26b5c277028d434cb11c54e9f5deec31"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-pswb"
server_key: "d556789bccfcd06ec00758c8d1b9e834"
server_key_iv: "644a750f35f56d812197f95cb3f7cf83"
server_key_version: "encryptionkey@0"
I20260620 23:13:08.361696 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "26b5c277028d434cb11c54e9f5deec31"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-pswb"
server_key: "d556789bccfcd06ec00758c8d1b9e834"
server_key_iv: "644a750f35f56d812197f95cb3f7cf83"
server_key_version: "encryptionkey@0"
I20260620 23:13:08.363272 22340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:13:08.364202 22355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:08.364435 22340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:08.364542 22340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "26b5c277028d434cb11c54e9f5deec31"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-pswb"
server_key: "d556789bccfcd06ec00758c8d1b9e834"
server_key_iv: "644a750f35f56d812197f95cb3f7cf83"
server_key_version: "encryptionkey@0"
I20260620 23:13:08.364632 22340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:08.381404 22340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:08.383317 22340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:08.383471 22340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:08.383688 22340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:08.384002 22340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:08.384055 22340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:08.384090 22340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:08.384119 22340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:08.390432 22340 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:37103
I20260620 23:13:08.390565 22468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:37103 every 8 connection(s)
I20260620 23:13:08.390844 22340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:08.393182 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 22340
I20260620 23:13:08.393290 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:13:08.393465 16700 external_mini_cluster.cc:1428] Setting key ff7c52b1e6d6fa44ea2d72e2fb93c21e
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:08.401675 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.391632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:45095 (local address 127.16.79.62:45027)
0620 23:13:08.391676 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.391678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.391831 (+ 153us) server_negotiation.cc:407] Connection header received
0620 23:13:08.392828 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.392829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.392839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.392902 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.393751 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.394502 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.396083 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.396408 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.396494 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.398859 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:08.398860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:08.398860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:08.398867 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:08.400219 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.400683 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.400684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.400684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.400791 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.401073 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.401073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.401073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.401166 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:08.401215 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.401546 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.401612 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:13:08.401921 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.391556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45027 (local address 127.16.79.1:45095)
0620 23:13:08.391706 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:08.391746 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:08.392645 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:08.392955 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:08.392958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:08.393353 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:08.393626 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.393631 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.394598 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.394599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.395887 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.395889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.396514 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.396515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.396751 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.397292 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:08.397300 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:08.398751 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:08.400305 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:08.400311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:08.400313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:08.400607 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:08.400859 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:08.400860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:08.400861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:08.401002 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:08.401237 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:08.401243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:08.401397 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 23:13:08.401516 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:13:08.401672 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260620 23:13:08.402422 22469 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45027
I20260620 23:13:08.402523 22469 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
I20260620 23:13:08.402815 22469 heartbeater.cc:511] Master 127.16.79.62:45027 requested a full tablet report, sending...
I20260620 23:13:08.403381 22278 ts_manager.cc:194] Registered new tserver with Master: 26b5c277028d434cb11c54e9f5deec31 (127.16.79.1:37103)
I20260620 23:13:08.404119 22278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:45095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:13:08.421422 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45027
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:08.502367 22474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:08.502547 22474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:08.502570 22474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:08.503621 22474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:08.503654 22474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:08.503679 22474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:08.505015 22474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:08.505175 22474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:08.505335 22474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:08.510088 22479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:08.510076 22482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:08.510210 22480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:08.510244 22474 server_base.cc:1034] running on GCE node
I20260620 23:13:08.510506 22474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:08.510739 22474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:08.511893 22474 hybrid_clock.cc:648] HybridClock initialized: now 1781997188511883 us; error 59 us; skew 500 ppm
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:08.514364 22474 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:13:08.514763 22474 webserver.cc:460] Webserver started at http://127.16.79.2:42771/ using document root <none> and password file <none>
I20260620 23:13:08.514974 22474 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:08.515022 22474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:08.515149 22474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:08.516104 22474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "b346864b80784c45b6ef7e3d9141883e"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-pswb"
server_key: "e07fdbb0ce90f71aebbb48e48789bfd3"
server_key_iv: "0d272cc87f00da97c0a2684013468f8b"
server_key_version: "encryptionkey@0"
I20260620 23:13:08.516440 22474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "b346864b80784c45b6ef7e3d9141883e"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-pswb"
server_key: "e07fdbb0ce90f71aebbb48e48789bfd3"
server_key_iv: "0d272cc87f00da97c0a2684013468f8b"
server_key_version: "encryptionkey@0"
I20260620 23:13:08.517885 22474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:13:08.518694 22489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:08.518949 22474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:13:08.519018 22474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "b346864b80784c45b6ef7e3d9141883e"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-pswb"
server_key: "e07fdbb0ce90f71aebbb48e48789bfd3"
server_key_iv: "0d272cc87f00da97c0a2684013468f8b"
server_key_version: "encryptionkey@0"
I20260620 23:13:08.519114 22474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:08.555053 22474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:08.557260 22474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:08.557370 22474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:08.557586 22474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:08.557876 22474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:08.557928 22474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:08.557971 22474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:08.557996 22474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:08.563632 22474 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:34339
I20260620 23:13:08.563674 22602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:34339 every 8 connection(s)
I20260620 23:13:08.563956 22474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:13:08.567540 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 22474
I20260620 23:13:08.567637 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:08.567842 16700 external_mini_cluster.cc:1428] Setting key ca55f19ae4badd30c19162ceada395f9
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:08.574384 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.564730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:49137 (local address 127.16.79.62:45027)
0620 23:13:08.564780 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.564782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.565088 (+ 306us) server_negotiation.cc:407] Connection header received
0620 23:13:08.566361 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.566362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.566373 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.566427 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.566997 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.567750 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.568889 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.569204 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.569280 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.571635 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:08.571637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:08.571637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:08.571644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:08.573059 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.573484 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.573485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.573486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.573543 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.573784 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.573785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.573785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.573895 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:08.573969 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.574219 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.574295 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 23:13:08.574561 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.564573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45027 (local address 127.16.79.2:49137)
0620 23:13:08.564871 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:08.564939 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:08.566242 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:08.566480 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:08.566483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:08.566732 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:08.566924 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.566928 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.567834 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.567835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.568798 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.568800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.569326 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.569327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.569420 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.570001 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:08.570014 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:08.571511 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:08.573176 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:08.573182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:08.573184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:08.573394 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:08.573610 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:08.573611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:08.573611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:08.573714 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:08.573961 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:08.573966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:08.574096 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 23:13:08.574200 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:13:08.574346 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":120,"threads_started":1}
I20260620 23:13:08.575003 22603 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45027
I20260620 23:13:08.575088 22603 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:08.575265 22603 heartbeater.cc:511] Master 127.16.79.62:45027 requested a full tablet report, sending...
I20260620 23:13:08.575659 22278 ts_manager.cc:194] Registered new tserver with Master: b346864b80784c45b6ef7e3d9141883e (127.16.79.2:34339)
I20260620 23:13:08.576027 22278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:49137
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:13:08.592972 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45027
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:08.670926 22608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:08.671109 22608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:08.671131 22608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:08.672183 22608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:08.672220 22608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:08.672246 22608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:08.673626 22608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:08.673784 22608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:08.673941 22608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:08.678942 22616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:08.678952 22614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:08.678987 22613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:08.679298 22608 server_base.cc:1034] running on GCE node
I20260620 23:13:08.679522 22608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:08.679740 22608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:08.680908 22608 hybrid_clock.cc:648] HybridClock initialized: now 1781997188680901 us; error 34 us; skew 500 ppm
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:08.683308 22608 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:13:08.683751 22608 webserver.cc:460] Webserver started at http://127.16.79.3:41317/ using document root <none> and password file <none>
I20260620 23:13:08.683964 22608 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:08.684029 22608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:08.684180 22608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:08.685202 22608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "af046e0a507a40dfa1e356111faf9ba8"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-pswb"
server_key: "edb6230ab84a0950808944bb67b02ba7"
server_key_iv: "825038660bd26a97267c8cae3d646e7b"
server_key_version: "encryptionkey@0"
I20260620 23:13:08.685544 22608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "af046e0a507a40dfa1e356111faf9ba8"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-pswb"
server_key: "edb6230ab84a0950808944bb67b02ba7"
server_key_iv: "825038660bd26a97267c8cae3d646e7b"
server_key_version: "encryptionkey@0"
I20260620 23:13:08.686807 22608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:13:08.687525 22623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:08.687701 22608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:08.687794 22608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "af046e0a507a40dfa1e356111faf9ba8"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-pswb"
server_key: "edb6230ab84a0950808944bb67b02ba7"
server_key_iv: "825038660bd26a97267c8cae3d646e7b"
server_key_version: "encryptionkey@0"
I20260620 23:13:08.687877 22608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:08.698779 22608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:08.700423 22608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:08.700570 22608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:08.700817 22608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:08.701094 22608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:08.701148 22608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:08.701195 22608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:08.701222 22608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:08.706907 22608 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:44857
I20260620 23:13:08.706956 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:44857 every 8 connection(s)
I20260620 23:13:08.707244 22608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:08.715638 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.707876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:34391 (local address 127.16.79.62:45027)
0620 23:13:08.707944 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.707947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.708114 (+ 167us) server_negotiation.cc:407] Connection header received
0620 23:13:08.709009 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.709010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.709020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.709072 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.709586 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.710061 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.710950 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.711125 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.711178 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.713068 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:08.713069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:08.713069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:08.713075 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:08.714423 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.714765 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.714766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.714766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.714822 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.715017 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.715017 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.715018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.715108 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:08.715160 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.715484 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.715571 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:13:08.715806 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.707848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45027 (local address 127.16.79.3:34391)
0620 23:13:08.708005 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:08.708037 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:08.708924 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:08.709116 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:08.709119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:08.709347 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:08.709513 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.709517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.710139 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.710140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.710880 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.710881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.711205 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.711206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.711293 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.711667 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:08.711675 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:08.712989 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:08.714500 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:08.714505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:08.714506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:08.714703 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:08.714880 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:08.714880 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:08.714881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:08.714967 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:08.715173 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:08.715179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:08.715333 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 23:13:08.715441 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:13:08.715598 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260620 23:13:08.716218 22737 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45027
I20260620 23:13:08.716305 22737 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:08.716475 22737 heartbeater.cc:511] Master 127.16.79.62:45027 requested a full tablet report, sending...
I20260620 23:13:08.716938 22278 ts_manager.cc:194] Registered new tserver with Master: af046e0a507a40dfa1e356111faf9ba8 (127.16.79.3:44857)
I20260620 23:13:08.717110 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 22608
I20260620 23:13:08.717173 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:08.717384 16700 external_mini_cluster.cc:1428] Setting key c79c09209260237aaaa36e914d9a018d
I20260620 23:13:08.717413 22278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:34391
I20260620 23:13:08.718261 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/test-admin.keytab.
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:08.733521 16700 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 23:13:08.735014 22744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:08.734281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.16.79.1:37103)
0620 23:13:08.734446 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.734450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.734458 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:13:08.734481 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.734482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.734494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.734587 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.734938 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48292: BlockingRecv error: recv got EOF from 127.0.0.1:48292 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
W20260620 23:13:08.735189 16700 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:08.736101 16700 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:08.737469 16700 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.1@KRBTEST.COM
I20260620 23:13:08.745357 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.738141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.16.79.1:37103)
0620 23:13:08.738214 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.738216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.738276 (+ 60us) server_negotiation.cc:407] Connection header received
0620 23:13:08.738313 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.738314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.738323 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.738369 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.738776 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.739466 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.740199 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.740483 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.740538 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.742565 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:08.742571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:08.742571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:08.742577 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:08.744207 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.744555 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.744558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.744558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.744615 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.744857 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.744858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.744858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.744972 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:08.745065 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.745142 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.745199 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 23:13:08.747022 22744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:08.746453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.16.79.1:37103)
0620 23:13:08.746515 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.746517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.746591 (+ 74us) server_negotiation.cc:407] Connection header received
0620 23:13:08.746661 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.746662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.746671 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.746714 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.746979 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48304: BlockingRecv error: recv got EOF from 127.0.0.1:48304 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 23:13:08.747110 16700 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:08.747989 16700 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:08.748850 16700 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.1@KRBTEST.COM
I20260620 23:13:08.755685 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.749366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.16.79.1:37103)
0620 23:13:08.749400 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.749402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.749424 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:13:08.749469 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.749470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.749477 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.749520 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.749837 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.750281 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.751003 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.751256 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.751382 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.753332 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:08.753333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:08.753334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:08.753342 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:08.754640 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.754987 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.754988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.754988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.755038 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.755229 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.755229 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.755230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.755321 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:08.755415 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.755520 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.755593 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 23:13:08.757006 22744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:08.756524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.16.79.1:37103)
0620 23:13:08.756558 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.756559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.756613 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:13:08.756677 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.756678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.756684 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.756778 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.756980 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48326: BlockingRecv error: recv got EOF from 127.0.0.1:48326 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 23:13:08.757278 16700 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997163808439-16700-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:08.758121 16700 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:08.758980 16700 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.1@KRBTEST.COM
I20260620 23:13:08.765699 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.759466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.16.79.1:37103)
0620 23:13:08.759522 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.759523 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.759584 (+ 61us) server_negotiation.cc:407] Connection header received
0620 23:13:08.759680 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.759681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.759689 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.759743 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.760086 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.760520 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.761240 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.761425 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.761504 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.763381 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:08.763382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:08.763382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:08.763389 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:08.764648 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.765014 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.765015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.765015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.765065 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.765263 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.765264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.765265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.765349 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:08.765435 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.765550 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.765616 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:13:08.766263 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 22340
I20260620 23:13:08.771081 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 22474
I20260620 23:13:08.775027 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 22608
I20260620 23:13:08.779218 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 22246
2026-06-20T23:13:08Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1923 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22757](info): set up 2 sockets
Jun 20 23:13:08 dist-test-slave-pswb krb5kdc[22757](info): commencing operation
krb5kdc: starting...
W20260620 23:13:09.829033 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:13:09 dist-test-slave-pswb krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:13:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.62.keytab.
Entry for principal oryx/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.62.keytab.
I20260620 23:13:09.942854 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:41873
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.62.keytab
--principal=oryx/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:41873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:10.022517 22770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:10.022763 22770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:10.022801 22770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:10.023906 22770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:10.023975 22770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:10.023999 22770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:10.024026 22770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:10.024056 22770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:10.025503 22770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:41873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.62.keytab
--principal=oryx/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:10.025768 22770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:10.025983 22770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:10.030828 22778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:10.030858 22775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:10.030853 22776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:10.030900 22770 server_base.cc:1034] running on GCE node
I20260620 23:13:10.031267 22770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:10.031522 22770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:10.032750 22770 hybrid_clock.cc:648] HybridClock initialized: now 1781997190032726 us; error 40 us; skew 500 ppm
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:10.038735 22770 init.cc:377] Logged in from keytab as oryx/127.16.79.62@KRBTEST.COM (short username oryx)
I20260620 23:13:10.039216 22770 webserver.cc:460] Webserver started at http://127.16.79.62:39055/ using document root <none> and password file <none>
I20260620 23:13:10.039446 22770 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:10.039510 22770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:10.039666 22770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:10.040738 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "36ed237bf6274042870df583d008b757"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "652d2a5781f8ebecf54bed8774f7b80b"
server_key_iv: "28021a6cc6cd2dbc409ee2660cf28f57"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.041138 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "36ed237bf6274042870df583d008b757"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "652d2a5781f8ebecf54bed8774f7b80b"
server_key_iv: "28021a6cc6cd2dbc409ee2660cf28f57"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.042444 22770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:10.043188 22785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.043421 22770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:10.043493 22770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "36ed237bf6274042870df583d008b757"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "652d2a5781f8ebecf54bed8774f7b80b"
server_key_iv: "28021a6cc6cd2dbc409ee2660cf28f57"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.043578 22770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:10.076826 22770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:10.079450 22770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:10.079675 22770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:10.083963 22770 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:41873
I20260620 23:13:10.083992 22837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:41873 every 8 connection(s)
I20260620 23:13:10.084345 22770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:10.085315 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:10.087380 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 22770
I20260620 23:13:10.087471 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:10.087532 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757: Bootstrap starting.
I20260620 23:13:10.087677 16700 external_mini_cluster.cc:1428] Setting key 4f07007dabd2c1c6df61c7ad5edd9221
I20260620 23:13:10.088223 22838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:10.088537 22838 log.cc:824] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:10.089390 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757: No bootstrap required, opened a new log
I20260620 23:13:10.091552 22838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } }
I20260620 23:13:10.091709 22838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:10.091739 22838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ed237bf6274042870df583d008b757, State: Initialized, Role: FOLLOWER
I20260620 23:13:10.091862 22838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } }
I20260620 23:13:10.091948 22838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:10.091980 22838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:10.092036 22838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:10.092753 22838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } }
I20260620 23:13:10.092859 22838 leader_election.cc:302] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36ed237bf6274042870df583d008b757; no voters:
I20260620 23:13:10.093039 22838 leader_election.cc:288] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:10.093086 22843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:10.093284 22838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:10.093356 22843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 1 LEADER]: Becoming Leader. State: Replica: 36ed237bf6274042870df583d008b757, State: Running, Role: LEADER
I20260620 23:13:10.093482 22843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } }
I20260620 23:13:10.093869 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } }
I20260620 23:13:10.093888 22845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36ed237bf6274042870df583d008b757. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } }
I20260620 23:13:10.094100 22845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:10.094363 22852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:10.094017 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:10.094913 22852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
I20260620 23:13:10.096386 22852 catalog_manager.cc:1347] Generated new cluster ID: d50a1abbe97d4865bcba641d0523acfe
I20260620 23:13:10.096437 22852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:10.099033 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.088231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.16.79.62:41873)
0620 23:13:10.088442 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:10.088449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:10.088472 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:13:10.089112 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:10.089119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:10.089165 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:10.089277 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:10.090078 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.090952 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.092058 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.092647 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.092786 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.095672 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:10.095676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:10.095676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:10.095687 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:10.097573 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.097945 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.097948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.097948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.098039 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.098249 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.098250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.098250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.098476 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:10.098626 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:10.098736 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 23:13:10.098865 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260620 23:13:10.102918 22852 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:10.103586 22852 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:10.115547 22852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757: Generated new TSK 0
I20260620 23:13:10.115707 22852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab.
Entry for principal oryx/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab.
I20260620 23:13:10.140920 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:41873
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab
--principal=oryx/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:10.231146 22864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:10.231392 22864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:10.231432 22864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:10.232645 22864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:10.232735 22864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:10.232795 22864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:10.234241 22864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab
--principal=oryx/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:10.234493 22864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:10.234730 22864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:10.240473 22869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:10.240515 22870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:10.240536 22872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:10.240751 22864 server_base.cc:1034] running on GCE node
I20260620 23:13:10.241021 22864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:10.241302 22864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:10.242529 22864 hybrid_clock.cc:648] HybridClock initialized: now 1781997190242494 us; error 47 us; skew 500 ppm
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:10.246052 22864 init.cc:377] Logged in from keytab as oryx/127.16.79.1@KRBTEST.COM (short username oryx)
I20260620 23:13:10.246670 22864 webserver.cc:460] Webserver started at http://127.16.79.1:45637/ using document root <none> and password file <none>
I20260620 23:13:10.246948 22864 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:10.247022 22864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:10.247210 22864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:10.248395 22864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "58731d03947949188070b9cc06fd12a4"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "cd83ca63348595100a9b3f6f427f4f67"
server_key_iv: "385a25b2f5d2956715fc03b31cac99c6"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.248812 22864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "58731d03947949188070b9cc06fd12a4"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "cd83ca63348595100a9b3f6f427f4f67"
server_key_iv: "385a25b2f5d2956715fc03b31cac99c6"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.250402 22864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:10.251358 22879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.251662 22864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:10.251793 22864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "58731d03947949188070b9cc06fd12a4"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "cd83ca63348595100a9b3f6f427f4f67"
server_key_iv: "385a25b2f5d2956715fc03b31cac99c6"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.251914 22864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:10.260502 22864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:10.262610 22864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:10.262773 22864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:10.263000 22864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:10.263674 22864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:10.263792 22864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.263836 22864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:10.263880 22864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.272547 22864 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:46527
I20260620 23:13:10.272599 22992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:46527 every 8 connection(s)
I20260620 23:13:10.273187 22864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:10.277032 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 22864
I20260620 23:13:10.277113 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:13:10.277324 16700 external_mini_cluster.cc:1428] Setting key e7a9e0491eafbf3a20b115456855654d
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
I20260620 23:13:10.284253 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.274106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:38073 (local address 127.16.79.62:41873)
0620 23:13:10.274221 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:10.274224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:10.274403 (+ 179us) server_negotiation.cc:407] Connection header received
0620 23:13:10.274963 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:10.274964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:10.274975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:10.275027 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:10.275970 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.276467 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.278055 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.278298 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.278383 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.281145 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:10.281147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:10.281148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:10.281158 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:10.282723 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.283187 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.283188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.283189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.283252 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.283517 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.283518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.283519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.283660 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:10.283746 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:10.284050 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 23:13:10.284146 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 23:13:10.284495 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.274032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.1:38073)
0620 23:13:10.274267 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:10.274318 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:10.274858 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:10.275110 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:10.275119 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:10.275676 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:10.275868 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.275873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.276788 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.276790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.277934 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.277938 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.278404 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.278405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.278539 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.279179 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:10.279188 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:10.281030 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:10.282831 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.282839 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.282842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.283094 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.283321 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.283321 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.283322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.283443 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.283735 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:10.283742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:10.283904 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 23:13:10.284026 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 23:13:10.284211 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260620 23:13:10.285162 22993 heartbeater.cc:347] Connected to a master server at 127.16.79.62:41873
I20260620 23:13:10.285274 22993 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:10.285512 22993 heartbeater.cc:511] Master 127.16.79.62:41873 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.16.79.2@KRBTEST.COM; defaulting to no policy
I20260620 23:13:10.286067 22802 ts_manager.cc:194] Registered new tserver with Master: 58731d03947949188070b9cc06fd12a4 (127.16.79.1:46527)
I20260620 23:13:10.286721 22802 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:38073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab.
Entry for principal oryx/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab.
I20260620 23:13:10.304525 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:41873
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab
--principal=oryx/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:10.390484 22998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:10.390684 22998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:10.390708 22998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:10.392143 22998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:10.392189 22998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:10.392215 22998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:10.393985 22998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab
--principal=oryx/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:10.394203 22998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:10.394405 22998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:10.399471 23004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:10.399516 23003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:10.399633 23006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:10.399852 22998 server_base.cc:1034] running on GCE node
I20260620 23:13:10.400100 22998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:10.400364 22998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:10.401539 22998 hybrid_clock.cc:648] HybridClock initialized: now 1781997190401541 us; error 52 us; skew 500 ppm
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:10.404584 22998 init.cc:377] Logged in from keytab as oryx/127.16.79.2@KRBTEST.COM (short username oryx)
I20260620 23:13:10.405071 22998 webserver.cc:460] Webserver started at http://127.16.79.2:44209/ using document root <none> and password file <none>
I20260620 23:13:10.405305 22998 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:10.405355 22998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:10.405481 22998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:10.406519 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "edb8acd7a81a4495889f5a4c3c83a47d"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "7cc6a21ab5061c6f4f2cad49bb1449c1"
server_key_iv: "32b79b7a937d34bae92bb46e12ee2e4c"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.406873 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "edb8acd7a81a4495889f5a4c3c83a47d"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "7cc6a21ab5061c6f4f2cad49bb1449c1"
server_key_iv: "32b79b7a937d34bae92bb46e12ee2e4c"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.408288 22998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:13:10.409260 23013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.409547 22998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:13:10.409689 22998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "edb8acd7a81a4495889f5a4c3c83a47d"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "7cc6a21ab5061c6f4f2cad49bb1449c1"
server_key_iv: "32b79b7a937d34bae92bb46e12ee2e4c"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.409796 22998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:10.423401 22998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:10.425662 22998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:10.425805 22998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:10.426044 22998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:10.426353 22998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:10.426385 22998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.426409 22998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:10.426452 22998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.433794 22998 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:40199
I20260620 23:13:10.433866 23126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:40199 every 8 connection(s)
I20260620 23:13:10.434231 22998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:13:10.440264 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 22998
I20260620 23:13:10.440367 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:10.440600 16700 external_mini_cluster.cc:1428] Setting key 56ec88309f2c364565068763913e63eb
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
I20260620 23:13:10.444578 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.435090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:42663 (local address 127.16.79.62:41873)
0620 23:13:10.435189 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:10.435191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:10.435348 (+ 157us) server_negotiation.cc:407] Connection header received
0620 23:13:10.435880 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:10.435881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:10.435893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:10.435954 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:10.437015 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.437519 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.438488 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.438709 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.438796 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.441397 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:10.441398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:10.441399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:10.441406 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:10.443030 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.443592 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.443593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.443593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.443649 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.443957 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.443957 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.443958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.444058 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:10.444114 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:10.444445 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 23:13:10.444513 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:13:10.445168 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.434987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.2:42663)
0620 23:13:10.435230 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:10.435265 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:10.435773 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:10.436021 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:10.436024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:10.436669 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:10.436924 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.436928 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.437614 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.437615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.438397 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.438399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.438804 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.438805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.438927 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.439444 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:10.439452 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:10.441313 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:10.443112 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.443118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.443120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.443461 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.443744 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.443745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.443745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.443885 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.444127 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:10.444133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:10.444298 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 23:13:10.444418 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 23:13:10.444605 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":120,"threads_started":1}
I20260620 23:13:10.445727 23127 heartbeater.cc:347] Connected to a master server at 127.16.79.62:41873
I20260620 23:13:10.445875 23127 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:10.446067 23127 heartbeater.cc:511] Master 127.16.79.62:41873 requested a full tablet report, sending...
I20260620 23:13:10.446511 22802 ts_manager.cc:194] Registered new tserver with Master: edb8acd7a81a4495889f5a4c3c83a47d (127.16.79.2:40199)
I20260620 23:13:10.446882 22802 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:42663
WARNING: no policy specified for oryx/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab.
Entry for principal oryx/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab.
I20260620 23:13:10.469164 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:41873
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab
--principal=oryx/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:10.549944 23132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:10.550135 23132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:10.550158 23132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:10.551258 23132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:10.551299 23132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:10.551324 23132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:10.552745 23132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab
--principal=oryx/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:10.553013 23132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:10.553236 23132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:10.558072 23140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:10.558079 23137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:10.558085 23138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:10.558563 23132 server_base.cc:1034] running on GCE node
I20260620 23:13:10.558758 23132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:10.558979 23132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:10.560127 23132 hybrid_clock.cc:648] HybridClock initialized: now 1781997190560083 us; error 61 us; skew 500 ppm
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:10.562610 23132 init.cc:377] Logged in from keytab as oryx/127.16.79.3@KRBTEST.COM (short username oryx)
I20260620 23:13:10.563053 23132 webserver.cc:460] Webserver started at http://127.16.79.3:34345/ using document root <none> and password file <none>
I20260620 23:13:10.563271 23132 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:10.563339 23132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:10.563484 23132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:10.564541 23132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "886b0c79c739f865b40932b9284a09dd"
server_key_iv: "f620b4f65d4e29232e5b8b38429d181b"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.565197 23132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "886b0c79c739f865b40932b9284a09dd"
server_key_iv: "f620b4f65d4e29232e5b8b38429d181b"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.567337 23132 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 23:13:10.568169 23147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.568344 23132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:10.568449 23132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "886b0c79c739f865b40932b9284a09dd"
server_key_iv: "f620b4f65d4e29232e5b8b38429d181b"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.568531 23132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:10.597533 23132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:10.599871 23132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:10.600023 23132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:10.600258 23132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:10.600561 23132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:10.600615 23132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.600656 23132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:10.600682 23132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.606818 23132 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:44657
I20260620 23:13:10.606851 23260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:44657 every 8 connection(s)
I20260620 23:13:10.607164 23132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
I20260620 23:13:10.614984 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 23132
I20260620 23:13:10.615070 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:10.615249 16700 external_mini_cluster.cc:1428] Setting key a2412653ed13d24f9e231893026023f7
I20260620 23:13:10.616041 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.607866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:46423 (local address 127.16.79.62:41873)
0620 23:13:10.607903 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:10.607905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:10.608100 (+ 195us) server_negotiation.cc:407] Connection header received
0620 23:13:10.608559 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:10.608560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:10.608569 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:10.608625 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:10.609524 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.609996 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.611005 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.611181 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.611244 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.613171 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:10.613172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:10.613172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:10.613178 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:10.614412 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.614823 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.614824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.614824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.614880 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.615211 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.615211 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.615212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.615314 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:10.615402 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:10.615869 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 23:13:10.615957 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 23:13:10.616200 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.607831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.3:46423)
0620 23:13:10.607980 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:10.608020 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:10.608484 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:10.608672 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:10.608675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:10.609271 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:10.609451 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.609455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.610131 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.610132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.610913 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.610916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.611261 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.611262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.611359 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.611744 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:10.611751 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:10.613077 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:10.614511 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.614517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.614519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.614729 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.614955 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.614956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.614956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.615095 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.615504 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:10.615511 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:10.615698 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 23:13:10.615821 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 23:13:10.615983 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260620 23:13:10.616766 23261 heartbeater.cc:347] Connected to a master server at 127.16.79.62:41873
I20260620 23:13:10.616868 23261 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:10.617030 23261 heartbeater.cc:511] Master 127.16.79.62:41873 requested a full tablet report, sending...
I20260620 23:13:10.617367 22802 ts_manager.cc:194] Registered new tserver with Master: 5eb4a8ac81a14379b3ba1bec6b8e25b9 (127.16.79.3:44657)
I20260620 23:13:10.617723 22802 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:46423
I20260620 23:13:10.627137 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.79.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.61.keytab.
Entry for principal oryx/127.16.79.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.61.keytab.
I20260620 23:13:10.651176 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.61:39717
--webserver_interface=127.16.79.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.61.keytab
--principal=oryx/127.16.79.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:41873,127.16.79.61:39717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:10.730139 23266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:10.730324 23266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:10.730345 23266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:10.731453 23266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:10.731493 23266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:10.731508 23266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:10.731547 23266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:10.731575 23266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:10.732995 23266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:41873,127.16.79.61:39717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.61.keytab
--principal=oryx/127.16.79.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.61:39717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.79.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
W20260620 23:13:10.733081 23266 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.79.62:41873,127.16.79.61:39717'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 23:13:10.733256 23266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:10.733475 23266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:13:10.738078 23266 server_base.cc:1034] running on GCE node
W20260620 23:13:10.738088 23274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:10.738097 23271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:10.738077 23272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:10.738480 23266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:10.738754 23266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:10.739926 23266 hybrid_clock.cc:648] HybridClock initialized: now 1781997190739909 us; error 50 us; skew 500 ppm
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:10.742517 23266 init.cc:377] Logged in from keytab as oryx/127.16.79.61@KRBTEST.COM (short username oryx)
I20260620 23:13:10.742959 23266 webserver.cc:460] Webserver started at http://127.16.79.61:36043/ using document root <none> and password file <none>
I20260620 23:13:10.743142 23266 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:10.743180 23266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:10.743273 23266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:10.744176 23266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data/instance:
uuid: "c8d362c900e64b6d829a6ce9d0378a1b"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "881e7b4e58494ec18a5096962fc23c62"
server_key_iv: "d46a5dfb721dab0da747368fd3cc2be1"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.744458 23266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/wal/instance:
uuid: "c8d362c900e64b6d829a6ce9d0378a1b"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "881e7b4e58494ec18a5096962fc23c62"
server_key_iv: "d46a5dfb721dab0da747368fd3cc2be1"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.745731 23266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:13:10.746402 23281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.746570 23266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:10.746654 23266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/wal
uuid: "c8d362c900e64b6d829a6ce9d0378a1b"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "881e7b4e58494ec18a5096962fc23c62"
server_key_iv: "d46a5dfb721dab0da747368fd3cc2be1"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.746738 23266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:10.756500 23266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:10.758406 23266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:10.758558 23266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:10.762176 23266 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.61:39717
I20260620 23:13:10.762214 23333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.61:39717 every 8 connection(s)
I20260620 23:13:10.762542 23266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
I20260620 23:13:10.763475 23334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:10.764940 23334 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:13:10.765112 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 23266
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.61@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.79.61@KRBTEST.COM
I20260620 23:13:10.774646 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.765672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.16.79.62:41873)
0620 23:13:10.765752 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:10.765754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:10.765979 (+ 225us) server_negotiation.cc:407] Connection header received
0620 23:13:10.766645 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:10.766646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:10.766659 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:10.766738 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:10.767198 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.767712 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.768846 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.769136 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.769212 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.771476 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:10.771478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:10.771478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:10.771485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:10.773075 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.773601 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.773602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.773603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.773658 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.773918 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.773943 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.773943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.774128 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:10.774185 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:10.774469 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 23:13:10.774546 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:13:10.774799 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.765599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:58472)
0620 23:13:10.765831 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:10.765894 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:10.766477 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:10.766758 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:10.766760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:10.766977 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:10.767110 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.767114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.767807 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.767808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.768767 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.768769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.769236 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.769236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.769355 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.769847 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:10.769854 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:10.771404 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:10.773154 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.773160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.773163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.773508 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.773733 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.773734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.773734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.773846 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.774198 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:10.774203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:10.774342 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 23:13:10.774421 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:13:10.774554 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":23,"thread_start_us":101,"threads_started":1}
I20260620 23:13:10.775331 23334 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:13:10.775463 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.765904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.16.79.61:39717)
0620 23:13:10.766054 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:10.766057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:10.766129 (+ 72us) server_negotiation.cc:407] Connection header received
0620 23:13:10.766479 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:10.766487 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:10.766512 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:10.766605 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:10.767322 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.768346 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.769227 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.769637 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.769721 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.772414 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:10.772420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:10.772421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:10.772428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:10.774149 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.774536 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.774539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.774539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.774592 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.774810 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.774811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.774812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.775007 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:10.775172 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:10.775240 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 23:13:10.775321 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.61@KRBTEST.COM for oryx/127.16.79.61@KRBTEST.COM
I20260620 23:13:10.781088 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.775577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39717 (local address 127.0.0.1:38076)
0620 23:13:10.775613 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:10.775622 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:10.775654 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:10.775981 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:10.775982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:10.776133 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:10.776160 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.776161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.776835 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.776836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.777276 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.777277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.777611 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.777612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.777654 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.777964 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:10.777968 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:10.779010 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:10.780280 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.780281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.780281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.780437 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.780605 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.780606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.780606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.780653 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.780883 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:10.780884 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:10.780915 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:13:10.780969 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:13:10.781036 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 23:13:10.781147 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.775655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.16.79.61:39717)
0620 23:13:10.775866 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:10.775867 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:10.775874 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:10.775888 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:10.775888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:10.775895 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:10.775970 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:10.776233 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.776666 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.777353 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.777541 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.777596 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.779083 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:10.779084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:10.779084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:10.779089 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:10.780199 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.780503 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.780504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.780504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.780548 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.780734 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.780735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.780735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.780828 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:10.780876 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:10.780982 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 23:13:10.781078 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":198}
I20260620 23:13:10.782246 23334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: Bootstrap starting.
I20260620 23:13:10.782941 23334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:10.783145 23334 log.cc:824] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:10.783759 23334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: No bootstrap required, opened a new log
I20260620 23:13:10.785104 23334 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } }
I20260620 23:13:10.785214 23334 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:10.785236 23334 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d362c900e64b6d829a6ce9d0378a1b, State: Initialized, Role: FOLLOWER
I20260620 23:13:10.785346 23334 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } }
I20260620 23:13:10.785588 23340 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } } }
I20260620 23:13:10.785661 23334 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:10.785668 23340 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:13:10.787338 23351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:13:10.787407 23351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:13:10.788995 23266 master_runner.cc:428] Detected that this master c8d362c900e64b6d829a6ce9d0378a1b is joining an existing cluster
I20260620 23:13:10.789048 23266 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.79.61:39717
I20260620 23:13:10.796155 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.790242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:58488)
0620 23:13:10.790369 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:10.790378 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:10.790458 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:10.790590 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:10.790591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:10.790752 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:10.790797 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.790799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.791429 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.791430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.792032 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.792033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.792430 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.792431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.792493 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.792964 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:10.792969 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:10.793330 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:10.794982 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.794982 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.794983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.795209 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.795439 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.795439 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.795440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.795526 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.795804 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:10.795805 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:10.795839 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:13:10.795924 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:13:10.796053 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260620 23:13:10.796588 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.790161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.16.79.62:41873)
0620 23:13:10.790230 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:10.790232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:10.790446 (+ 214us) server_negotiation.cc:407] Connection header received
0620 23:13:10.790516 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:10.790517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:10.790527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:10.790569 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:10.790881 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.791348 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.792138 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.792336 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.792406 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.793415 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:10.793416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:10.793417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:10.793424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:10.794910 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.795302 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.795303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.795304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.795360 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.795604 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.795606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.795606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.795734 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:10.795813 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:10.795968 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 23:13:10.796044 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:13:10.798357 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.790115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38090 (local address 127.16.79.61:39717)
0620 23:13:10.790202 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:10.790203 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:10.790273 (+ 70us) server_negotiation.cc:407] Connection header received
0620 23:13:10.790361 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:10.790362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:10.790370 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:10.790395 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:10.790886 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.791581 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.792585 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.792939 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.793008 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.795316 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:10.795317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:10.795318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:10.795323 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:10.797041 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.797396 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.797397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.797398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.797446 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.797645 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.797646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.797646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.797766 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:10.797857 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:10.798144 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 23:13:10.798258 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":69}
I20260620 23:13:10.798673 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.790074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39717 (local address 127.0.0.1:38090)
0620 23:13:10.790208 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:10.790219 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:10.790295 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:10.790443 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:10.790444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:10.790667 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:10.790780 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.790782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.791666 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.791666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.792503 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.792505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.793021 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.793022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.793088 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.793693 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:10.793698 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:10.795233 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:10.797118 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.797119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.797119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.797325 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.797503 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.797504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.797505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.797565 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.797850 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:10.797850 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:10.798024 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 23:13:10.798120 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:13:10.798551 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.62@KRBTEST.COM for oryx/127.16.79.61@KRBTEST.COM
I20260620 23:13:10.805907 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.798725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38104 (local address 127.16.79.61:39717)
0620 23:13:10.798942 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:10.798944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:10.798951 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:10.798980 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:10.798981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:10.798990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:10.799039 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:10.799637 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.800062 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.800869 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.801061 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.801150 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.803378 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:10.803380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:10.803380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:10.803386 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:10.804784 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.805152 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.805153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.805153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.805203 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.805414 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.805415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.805416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.805517 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:10.805587 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:10.805741 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:13:10.805820 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260620 23:13:10.805928 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.798666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39717 (local address 127.0.0.1:38104)
0620 23:13:10.798828 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:10.798861 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:10.798917 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:10.799042 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:10.799045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:10.799489 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:10.799542 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.799544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.800147 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.800149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.800770 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.800772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.801152 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.801154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.801227 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.801652 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:10.801657 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:10.803235 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:10.804875 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.804882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.804884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.805078 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.805285 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.805286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.805286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.805343 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.805581 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:10.805587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:10.805638 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:13:10.805719 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:13:10.805821 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":58,"threads_started":1}
I20260620 23:13:10.806664 22802 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.79.61:39717: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b"
member_type: NON_VOTER
last_known_addr {
host: "127.16.79.61"
port: 39717
}
attrs {
promote: true
}
}
dest_uuid: "36ed237bf6274042870df583d008b757"
cas_config_opid_index: -1
I20260620 23:13:10.806970 22802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true } }
I20260620 23:13:10.807420 23360 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c8d362c900e64b6d829a6ce9d0378a1b (127.16.79.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true } } }
I20260620 23:13:10.807714 23308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:10.808327 23360 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.79.61:39717
I20260620 23:13:10.808413 23361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true } } }
I20260620 23:13:10.808485 23361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:10.808573 23308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 FOLLOWER]: Refusing update from remote peer 36ed237bf6274042870df583d008b757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 23:13:10.808950 23266 master.cc:537] Master@127.16.79.61:39717 shutting down...
I20260620 23:13:10.808957 23361 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:13:10.809633 22786 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 23:13:10.809995 22786 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 -> Peer c8d362c900e64b6d829a6ce9d0378a1b (127.16.79.61:39717): Couldn't send request to peer c8d362c900e64b6d829a6ce9d0378a1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 23:13:10.811882 23266 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:10.812086 23266 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:10.812199 23266 tablet_replica.cc:330] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: stopping tablet replica
I20260620 23:13:10.824680 23266 master.cc:552] Master@127.16.79.61:39717 shutdown complete.
I20260620 23:13:10.824787 23266 master_runner.cc:305] Clearing existing system tablet
I20260620 23:13:10.824929 23266 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:10.824966 23266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:10.825598 23266 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:10.826200 23365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.826330 23266 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:10.826378 23266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:10.826406 23266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/wal
uuid: "c8d362c900e64b6d829a6ce9d0378a1b"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "881e7b4e58494ec18a5096962fc23c62"
server_key_iv: "d46a5dfb721dab0da747368fd3cc2be1"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.827415 23266 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:10.828751 23266 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:13:10.828830 23266 log.cc:1196] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 23:13:10.829051 23266 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: Deleting consensus metadata
I20260620 23:13:10.829181 23266 master_runner.cc:315] Copying system tablet from 127.16.79.62:41873
I20260620 23:13:10.829736 23266 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.79.62:41873
I20260620 23:13:10.835745 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.830096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.16.79.62:41873)
0620 23:13:10.830161 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:10.830164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:10.830266 (+ 102us) server_negotiation.cc:407] Connection header received
0620 23:13:10.830367 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:10.830368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:10.830378 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:10.830449 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:10.830765 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.831245 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.832005 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.832193 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.832267 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.833212 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:10.833213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:10.833213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:10.833220 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:10.834621 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.834981 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.834982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.834983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.835040 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.835243 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.835244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.835244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.835349 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:10.835425 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:10.835574 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:13:10.835656 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:13:10.835757 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.829994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:58500)
0620 23:13:10.830191 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:10.830201 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:10.830283 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:10.830451 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:10.830452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:10.830623 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:10.830685 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.830687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.831329 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.831330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.831920 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:10.831921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.832265 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.832266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:10.832343 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.832853 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:10.832857 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:10.833123 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:10.834731 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.834732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.834732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.834907 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.835113 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:10.835114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:10.835114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:10.835180 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:10.835417 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:10.835418 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:10.835462 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 23:13:10.835550 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:13:10.835649 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260620 23:13:10.836258 22822 tablet_copy_service.cc:140] P 36ed237bf6274042870df583d008b757: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c8d362c900e64b6d829a6ce9d0378a1b ({username='oryx', principal='oryx/127.16.79.61@KRBTEST.COM'} at 127.0.0.1:58500)
I20260620 23:13:10.836354 22822 tablet_copy_service.cc:161] P 36ed237bf6274042870df583d008b757: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c8d362c900e64b6d829a6ce9d0378a1b at {username='oryx', principal='oryx/127.16.79.61@KRBTEST.COM'} at 127.0.0.1:58500: session id = c8d362c900e64b6d829a6ce9d0378a1b-00000000000000000000000000000000
I20260620 23:13:10.837157 22822 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:13:10.837734 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:13:10.838562 23266 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: tablet copy: Starting download of 0 data blocks...
I20260620 23:13:10.838721 23266 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: tablet copy: Starting download of 1 WAL segments...
I20260620 23:13:10.839624 23266 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:13:10.840365 22822 tablet_copy_service.cc:342] P 36ed237bf6274042870df583d008b757: Request end of tablet copy session c8d362c900e64b6d829a6ce9d0378a1b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.79.61@KRBTEST.COM'} at 127.0.0.1:58500
I20260620 23:13:10.840411 22822 tablet_copy_service.cc:433] P 36ed237bf6274042870df583d008b757: ending tablet copy session c8d362c900e64b6d829a6ce9d0378a1b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c8d362c900e64b6d829a6ce9d0378a1b
I20260620 23:13:10.842646 23266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:10.843696 23373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:10.843760 23375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:10.843784 23372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:10.844852 23266 server_base.cc:1034] running on GCE node
I20260620 23:13:10.844946 23266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:10.845069 23266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:10.846189 23266 hybrid_clock.cc:648] HybridClock initialized: now 1781997190846171 us; error 30 us; skew 500 ppm
Jun 20 23:13:10 dist-test-slave-pswb krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:10.847435 23266 init.cc:377] Logged in from keytab as oryx/127.16.79.61@KRBTEST.COM (short username oryx)
I20260620 23:13:10.847862 23266 webserver.cc:460] Webserver started at http://127.16.79.61:46773/ using document root <none> and password file <none>
I20260620 23:13:10.848086 23266 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:10.848137 23266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:10.848937 23266 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:10.849551 23382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:10.849658 23266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:10.849697 23266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/wal
uuid: "c8d362c900e64b6d829a6ce9d0378a1b"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "881e7b4e58494ec18a5096962fc23c62"
server_key_iv: "d46a5dfb721dab0da747368fd3cc2be1"
server_key_version: "encryptionkey@0"
I20260620 23:13:10.849756 23266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:10.874039 23266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:10.875789 23266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:10.878760 23266 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.61:39717
I20260620 23:13:10.878809 23434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.61:39717 every 8 connection(s)
I20260620 23:13:10.879084 23266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
I20260620 23:13:10.879877 23435 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:13:10.880395 23435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: Bootstrap starting.
I20260620 23:13:10.883208 23435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:13:10.883445 23435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: Bootstrap complete.
I20260620 23:13:10.883651 23435 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true } }
I20260620 23:13:10.883754 23435 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d362c900e64b6d829a6ce9d0378a1b, State: Initialized, Role: LEARNER
I20260620 23:13:10.883793 23435 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true } }
I20260620 23:13:10.884084 23435 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:10.884109 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true } } }
I20260620 23:13:10.884218 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: This master's current role is: LEARNER
I20260620 23:13:10.885882 23449 catalog_manager.cc:1259] Loaded cluster ID: d50a1abbe97d4865bcba641d0523acfe
I20260620 23:13:10.885977 23449 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: loading cluster ID for follower catalog manager: success
I20260620 23:13:10.886741 23449 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: acquiring CA information for follower catalog manager: success
I20260620 23:13:10.886996 23449 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:13:10.933462 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:10.927419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.16.79.61:39717)
0620 23:13:10.927551 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:10.927553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:10.927561 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:13:10.927584 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:10.927584 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:10.927593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:10.927655 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:10.928267 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.928941 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.929633 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:10.929819 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:10.929876 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:10.931006 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:10.931007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:10.931007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:10.931013 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:10.932520 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.932863 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.932864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.932864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.932911 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:10.933053 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:10.933054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:10.933054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:10.933139 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:10.933204 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:10.933293 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 23:13:10.933347 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.16.79.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.60.keytab.
Entry for principal oryx/127.16.79.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.60.keytab.
I20260620 23:13:10.959717 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.60:46161
--webserver_interface=127.16.79.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.60.keytab
--principal=oryx/127.16.79.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:41873,127.16.79.61:39717,127.16.79.60:46161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:11.040900 23453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:11.041079 23453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:11.041100 23453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:11.042420 23453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:11.042469 23453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:11.042484 23453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:11.042497 23453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:11.042510 23453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:11.044199 23453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:41873,127.16.79.61:39717,127.16.79.60:46161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.60.keytab
--principal=oryx/127.16.79.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.60:46161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.79.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:11.044381 23453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:11.044533 23453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:11.049355 23459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:11.049352 23461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:11.049475 23458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:11.049511 23453 server_base.cc:1034] running on GCE node
I20260620 23:13:11.049846 23453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:11.050122 23453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:11.051290 23453 hybrid_clock.cc:648] HybridClock initialized: now 1781997191051274 us; error 23 us; skew 500 ppm
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:11.053747 23453 init.cc:377] Logged in from keytab as oryx/127.16.79.60@KRBTEST.COM (short username oryx)
I20260620 23:13:11.054195 23453 webserver.cc:460] Webserver started at http://127.16.79.60:37195/ using document root <none> and password file <none>
I20260620 23:13:11.054414 23453 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:11.054458 23453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:11.054590 23453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:11.055658 23453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data/instance:
uuid: "f235a868911c4de5b05381f909d97a75"
format_stamp: "Formatted at 2026-06-20 23:13:11 on dist-test-slave-pswb"
server_key: "bb13c4b9f8fe405480a6f4070554c19d"
server_key_iv: "7eb49cb7beb6302e9e2471c61ca941a3"
server_key_version: "encryptionkey@0"
I20260620 23:13:11.056030 23453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/wal/instance:
uuid: "f235a868911c4de5b05381f909d97a75"
format_stamp: "Formatted at 2026-06-20 23:13:11 on dist-test-slave-pswb"
server_key: "bb13c4b9f8fe405480a6f4070554c19d"
server_key_iv: "7eb49cb7beb6302e9e2471c61ca941a3"
server_key_version: "encryptionkey@0"
I20260620 23:13:11.057385 23453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:13:11.058115 23468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.058292 23453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:11.058400 23453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/wal
uuid: "f235a868911c4de5b05381f909d97a75"
format_stamp: "Formatted at 2026-06-20 23:13:11 on dist-test-slave-pswb"
server_key: "bb13c4b9f8fe405480a6f4070554c19d"
server_key_iv: "7eb49cb7beb6302e9e2471c61ca941a3"
server_key_version: "encryptionkey@0"
I20260620 23:13:11.058476 23453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:11.071112 23453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:11.073800 23453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:11.074051 23453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:11.078670 23453 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.60:46161
I20260620 23:13:11.078702 23520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.60:46161 every 8 connection(s)
I20260620 23:13:11.079039 23453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
I20260620 23:13:11.079938 23521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:11.081624 23521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:13:11.084281 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 23453
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.60@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.79.60@KRBTEST.COM
I20260620 23:13:11.091683 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.082467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.16.79.62:41873)
0620 23:13:11.082539 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.082541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.082644 (+ 103us) server_negotiation.cc:407] Connection header received
0620 23:13:11.083048 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.083049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.083059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.083116 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.083693 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.084441 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.086034 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.086234 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.086320 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.088437 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.088438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.088438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.088445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.089996 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.090415 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.090416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.090417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.090475 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.090919 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.090920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.090920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.091083 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.091161 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.091544 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.091622 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:13:11.091923 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.082293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:58516)
0620 23:13:11.082500 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.082544 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.082962 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.083185 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.083188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.083441 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.083613 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.083617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.084563 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.084563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.085950 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.085952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.086346 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.086347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.086466 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.086936 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.086943 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.088336 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.090081 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.090087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.090089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.090323 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.090742 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.090743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.090743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.090837 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.091199 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.091207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.091374 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.091510 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.091655 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":93,"threads_started":1}
I20260620 23:13:11.092541 23521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:13:11.093473 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.084750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.16.79.60:46161)
0620 23:13:11.084925 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.084929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.084944 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:13:11.084965 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.084967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.084990 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.085067 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.085754 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.086702 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.087403 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.087687 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.087762 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.090339 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.090345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.090346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.090353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.092081 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.092479 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.092482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.092482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.092545 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.092787 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.092788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.092788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.092966 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.093096 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.093172 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.093330 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.60@KRBTEST.COM for oryx/127.16.79.61@KRBTEST.COM
I20260620 23:13:11.099512 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.092840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39717 (local address 127.0.0.1:38148)
0620 23:13:11.092884 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.092894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.092944 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.093173 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.093174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.093365 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.093397 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.093398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.094160 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.094161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.094784 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.094786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.095162 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.095162 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.095218 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.095621 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.095626 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.096906 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.098472 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.098472 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.098473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.098637 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.098874 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.098875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.098875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.098932 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.099281 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.099281 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.099324 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.099388 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.099458 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 23:13:11.099583 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.092973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.16.79.61:39717)
0620 23:13:11.093018 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.093020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.093028 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:13:11.093049 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.093050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.093062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.093112 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.093469 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.094065 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.094861 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.095083 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.095160 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.096968 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.096969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.096970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.096976 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.098392 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.098723 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.098723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.098724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.098772 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.099027 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.099029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.099029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.099205 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.099269 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.099423 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.099494 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:13:11.100047 23521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.60@KRBTEST.COM for oryx/127.16.79.60@KRBTEST.COM
I20260620 23:13:11.106335 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.100436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46161 (local address 127.0.0.1:47328)
0620 23:13:11.100506 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.100520 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.100574 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.100964 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.100965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.101071 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.101102 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.101104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.101754 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.101755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.102223 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.102225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.102625 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.102626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.102666 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.103099 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.103103 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.104117 (+ 1014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.105520 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.105520 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.105521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.105672 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.105871 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.105872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.105872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.105930 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.106131 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.106131 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.106164 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.106214 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.106283 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 23:13:11.106448 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.100528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.16.79.60:46161)
0620 23:13:11.100647 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.100649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.100785 (+ 136us) server_negotiation.cc:407] Connection header received
0620 23:13:11.100856 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.100857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.100871 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.100991 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.101225 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.101674 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.102339 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.102539 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.102599 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.104188 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.104189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.104189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.104195 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.105408 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.105743 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.105743 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.105744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.105794 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.105988 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.105989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.105989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.106070 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.106115 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.106228 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.106323 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":88}
I20260620 23:13:11.107590 23521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: Bootstrap starting.
I20260620 23:13:11.108346 23521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:11.108618 23521 log.cc:824] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:11.109303 23521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: No bootstrap required, opened a new log
I20260620 23:13:11.110612 23521 raft_consensus.cc:348] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } }
I20260620 23:13:11.110719 23521 raft_consensus.cc:374] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:11.110740 23521 raft_consensus.cc:729] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f235a868911c4de5b05381f909d97a75, State: Initialized, Role: FOLLOWER
I20260620 23:13:11.110875 23521 consensus_queue.cc:260] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } }
I20260620 23:13:11.111166 23521 sys_catalog.cc:564] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:11.111145 23526 sys_catalog.cc:455] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } } }
I20260620 23:13:11.111238 23526 sys_catalog.cc:458] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:13:11.113042 23537 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:13:11.113101 23537 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:13:11.115089 23453 master_runner.cc:428] Detected that this master f235a868911c4de5b05381f909d97a75 is joining an existing cluster
I20260620 23:13:11.115159 23453 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.79.60:46161
I20260620 23:13:11.122902 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.116266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46161 (local address 127.0.0.1:47340)
0620 23:13:11.116558 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.116566 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.116624 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.116807 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.116808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.117024 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.117075 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.117076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.117915 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.117915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.118777 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.118779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.119254 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.119255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.119309 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.119894 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.119899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.120238 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.121967 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.121968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.121969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.122151 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.122360 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.122360 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.122361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.122409 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.122657 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.122657 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.122700 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.122764 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.122837 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260620 23:13:11.123010 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.116388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.16.79.60:46161)
0620 23:13:11.116420 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.116421 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.116607 (+ 186us) server_negotiation.cc:407] Connection header received
0620 23:13:11.116722 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.116723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.116732 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.116760 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.117147 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.117831 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.118867 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.119141 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.119177 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.120334 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.120335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.120335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.120341 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.121885 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.122220 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.122221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.122221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.122287 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.122470 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.122471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.122472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.122589 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.122657 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.122806 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.122897 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":18}
I20260620 23:13:11.141185 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.116103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.16.79.62:41873)
0620 23:13:11.116152 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.116154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.116310 (+ 156us) server_negotiation.cc:407] Connection header received
0620 23:13:11.116562 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.116563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.116576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.116607 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.117145 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.117859 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.118883 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.119184 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.119220 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.137724 (+ 18504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.137725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.137726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.137732 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.139857 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.140252 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.140253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.140254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.140314 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.140521 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.140522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.140522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.140668 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.140779 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.140990 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.141087 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:13:11.141245 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.116292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.16.79.61:39717)
0620 23:13:11.116324 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.116326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.116468 (+ 142us) server_negotiation.cc:407] Connection header received
0620 23:13:11.116522 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.116523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.116533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.116588 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.117050 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.117787 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.118939 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.119219 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.119258 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.138291 (+ 19033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.138294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.138294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.138306 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.140075 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.140502 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.140503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.140504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.140556 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.140766 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.140767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.140767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.140877 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.140953 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.141090 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.141160 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:13:11.141477 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.116077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:58526)
0620 23:13:11.116248 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.116259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.116335 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.116671 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.116672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.116982 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.117042 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.117044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.117943 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.117944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.118801 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.118803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.119286 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.119286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.119337 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.137308 (+ 17971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.137312 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.137603 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.139958 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.139960 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.139960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.140188 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.140382 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.140383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.140383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.140441 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.140811 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.140812 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.140870 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.141274 (+ 404us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.141377 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":26,"thread_start_us":68,"threads_started":1}
I20260620 23:13:11.141947 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.116128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39717 (local address 127.0.0.1:38158)
0620 23:13:11.116420 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.116429 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.116476 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.116634 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.116634 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.116889 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.116957 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.116959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.117883 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.117884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.118843 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.118845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.119343 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.119343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.119405 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.137653 (+ 18248us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.137658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.138011 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.140170 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.140171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.140171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.140399 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.140639 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.140639 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.140640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.140714 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.140946 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.140947 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.140988 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.141048 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.141160 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":48,"thread_start_us":72,"threads_started":1}
I20260620 23:13:11.144635 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 22864
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997190, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.62@KRBTEST.COM for oryx/127.16.79.60@KRBTEST.COM
I20260620 23:13:11.149925 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:46527
--local_ip_for_outbound_sockets=127.16.79.1
--tserver_master_addrs=127.16.79.62:41873
--webserver_port=45637
--webserver_interface=127.16.79.1
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab
--principal=oryx/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.79.62:41873,127.16.79.61:39717
--tserver_master_addrs=127.16.79.62:41873,127.16.79.61:39717,127.16.79.60:46161 with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:13:11.150352 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.143541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46161 (local address 127.0.0.1:47354)
0620 23:13:11.143580 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.143590 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.143627 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.143767 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.143768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.143893 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.143927 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.143928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.144800 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.144800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.145455 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.145457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.145846 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.145848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.145913 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.146314 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.146319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.147514 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.149251 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.149252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.149253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.149484 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.149707 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.149708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.149708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.149801 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.150082 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.150083 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.150133 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.150193 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.150293 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 23:13:11.150667 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.143613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.16.79.60:46161)
0620 23:13:11.143650 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.143652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.143661 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:11.143691 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.143691 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.143700 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.143746 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.143991 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.144686 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.145527 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.145758 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.145844 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.147598 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.147599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.147600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.147607 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.149162 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.149564 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.149565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.149566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.149617 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.149851 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.149852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.149853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.149948 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.150050 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.150540 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.150606 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:13:11.151511 22802 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.79.60:46161: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f235a868911c4de5b05381f909d97a75"
member_type: NON_VOTER
last_known_addr {
host: "127.16.79.60"
port: 46161
}
attrs {
promote: true
}
}
dest_uuid: "36ed237bf6274042870df583d008b757"
cas_config_opid_index: 5
I20260620 23:13:11.151731 22802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } }
I20260620 23:13:11.152086 23361 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f235a868911c4de5b05381f909d97a75 (127.16.79.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.152910 23361 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.79.60:46161
I20260620 23:13:11.152899 23495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:11.152992 23361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.153066 23361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:11.153110 23453 master.cc:537] Master@127.16.79.60:46161 shutting down...
I20260620 23:13:11.153811 23495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 FOLLOWER]: Refusing update from remote peer 36ed237bf6274042870df583d008b757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 23:13:11.154093 23360 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:13:11.154737 22788 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 -> Peer f235a868911c4de5b05381f909d97a75 (127.16.79.60:46161): Couldn't send request to peer f235a868911c4de5b05381f909d97a75. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 23:13:11.156283 23453 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:11.156368 23453 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:11.156418 23453 tablet_replica.cc:330] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: stopping tablet replica
I20260620 23:13:11.159161 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.154186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.16.79.61:39717)
0620 23:13:11.155848 (+ 1662us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.155850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.155857 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:11.155880 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.155881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.155902 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.155958 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:11.156193 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.156898 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.157839 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.158825 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.158918 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.158979 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.159073 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.159102 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1643}
I20260620 23:13:11.159183 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.154060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39717 (local address 127.0.0.1:38164)
0620 23:13:11.155718 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.155729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.155773 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.156011 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.156012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.156013 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:11.156048 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.156049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.156990 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.156990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.157747 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.157749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.158928 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.158929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.158991 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.159009 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.159066 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.159101 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1641}
I20260620 23:13:11.159492 23409 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 LEARNER]: Refusing update from remote peer 36ed237bf6274042870df583d008b757: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 23:13:11.159690 23360 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 23:13:11.160241 23409 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f235a868911c4de5b05381f909d97a75 (127.16.79.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.161525 23360 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757: attempting to promote NON_VOTER c8d362c900e64b6d829a6ce9d0378a1b to VOTER
I20260620 23:13:11.161542 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36ed237bf6274042870df583d008b757. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.161614 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: This master's current role is: LEARNER
I20260620 23:13:11.161648 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: NON_VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: true } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.161682 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: This master's current role is: LEARNER
I20260620 23:13:11.161710 23360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } }
I20260620 23:13:11.162170 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.162077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46161 (local address 127.0.0.1:47364)
0620 23:13:11.162106 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.162144 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 23:13:11.162199 23409 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 LEARNER]: Refusing update from remote peer 36ed237bf6274042870df583d008b757: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 23:13:11.162369 23361 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 23:13:11.162603 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.162518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46161 (local address 127.0.0.1:47366)
0620 23:13:11.162547 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.162577 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 23:13:11.162773 22788 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 -> Peer f235a868911c4de5b05381f909d97a75 (127.16.79.60:46161): Couldn't send request to peer f235a868911c4de5b05381f909d97a75. Status: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:13:11.162994 23549 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8d362c900e64b6d829a6ce9d0378a1b (127.16.79.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.163120 23409 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8d362c900e64b6d829a6ce9d0378a1b (127.16.79.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.163872 23548 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.163951 23548 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:13:11.164211 23361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.164317 23361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:11.164405 23361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.164446 23361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:11.169606 23453 master.cc:552] Master@127.16.79.60:46161 shutdown complete.
I20260620 23:13:11.169669 23453 master_runner.cc:305] Clearing existing system tablet
I20260620 23:13:11.169808 23453 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:11.169865 23453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:11.170449 23453 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:11.170930 23552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.171079 23453 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:11.171135 23453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.171165 23453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/wal
uuid: "f235a868911c4de5b05381f909d97a75"
format_stamp: "Formatted at 2026-06-20 23:13:11 on dist-test-slave-pswb"
server_key: "bb13c4b9f8fe405480a6f4070554c19d"
server_key_iv: "7eb49cb7beb6302e9e2471c61ca941a3"
server_key_version: "encryptionkey@0"
I20260620 23:13:11.171838 23453 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:11.173346 23453 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:13:11.173422 23453 log.cc:1196] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 23:13:11.173615 23453 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: Deleting consensus metadata
I20260620 23:13:11.173749 23453 master_runner.cc:315] Copying system tablet from 127.16.79.62:41873
I20260620 23:13:11.174371 23453 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: tablet copy: Beginning tablet copy session from remote peer at address 127.16.79.62:41873
I20260620 23:13:11.180806 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.174668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.16.79.62:41873)
0620 23:13:11.174706 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.174708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.174825 (+ 117us) server_negotiation.cc:407] Connection header received
0620 23:13:11.174909 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.174910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.174918 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.174958 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.175252 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.175709 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.176616 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.176926 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.176969 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.178066 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.178068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.178068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.178075 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.179692 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.180039 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.180040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.180040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.180097 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.180295 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.180296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.180297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.180400 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.180474 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.180617 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.180720 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:13:11.180835 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.174621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:58540)
0620 23:13:11.174750 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.174760 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.174851 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.174970 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.174971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.175133 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.175171 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.175172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.175786 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.175788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.176518 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.176521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.177054 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.177054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.177119 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.177583 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.177589 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.177968 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.179785 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.179786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.179787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.179952 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.180165 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.180166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.180167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.180222 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.180467 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.180468 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.180510 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.180593 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.180690 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260620 23:13:11.181356 22822 tablet_copy_service.cc:140] P 36ed237bf6274042870df583d008b757: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f235a868911c4de5b05381f909d97a75 ({username='oryx', principal='oryx/127.16.79.60@KRBTEST.COM'} at 127.0.0.1:58540)
I20260620 23:13:11.181409 22822 tablet_copy_service.cc:161] P 36ed237bf6274042870df583d008b757: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f235a868911c4de5b05381f909d97a75 at {username='oryx', principal='oryx/127.16.79.60@KRBTEST.COM'} at 127.0.0.1:58540: session id = f235a868911c4de5b05381f909d97a75-00000000000000000000000000000000
I20260620 23:13:11.182086 22822 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:13:11.182541 23453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:13:11.183359 23453 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: tablet copy: Starting download of 0 data blocks...
I20260620 23:13:11.183503 23453 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: tablet copy: Starting download of 1 WAL segments...
I20260620 23:13:11.184125 23453 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:13:11.184923 22822 tablet_copy_service.cc:342] P 36ed237bf6274042870df583d008b757: Request end of tablet copy session f235a868911c4de5b05381f909d97a75-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.79.60@KRBTEST.COM'} at 127.0.0.1:58540
I20260620 23:13:11.184975 22822 tablet_copy_service.cc:433] P 36ed237bf6274042870df583d008b757: ending tablet copy session f235a868911c4de5b05381f909d97a75-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f235a868911c4de5b05381f909d97a75
I20260620 23:13:11.187074 23453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:11.188089 23559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:11.188272 23562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:11.188745 23453 server_base.cc:1034] running on GCE node
W20260620 23:13:11.193769 23560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:11.194084 23453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:11.194312 23453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:11.195510 23453 hybrid_clock.cc:648] HybridClock initialized: now 1781997191195508 us; error 90 us; skew 500 ppm
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:11.197093 23453 init.cc:377] Logged in from keytab as oryx/127.16.79.60@KRBTEST.COM (short username oryx)
I20260620 23:13:11.197382 23453 webserver.cc:460] Webserver started at http://127.16.79.60:34327/ using document root <none> and password file <none>
I20260620 23:13:11.197495 23453 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:11.197530 23453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:11.198125 23453 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:11.198649 23569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.198837 23453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.198886 23453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/wal
uuid: "f235a868911c4de5b05381f909d97a75"
format_stamp: "Formatted at 2026-06-20 23:13:11 on dist-test-slave-pswb"
server_key: "bb13c4b9f8fe405480a6f4070554c19d"
server_key_iv: "7eb49cb7beb6302e9e2471c61ca941a3"
server_key_version: "encryptionkey@0"
I20260620 23:13:11.198944 23453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:11.207195 23453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:11.208920 23453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:11.211901 23453 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.60:46161
I20260620 23:13:11.211922 23621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.60:46161 every 8 connection(s)
I20260620 23:13:11.212195 23453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
I20260620 23:13:11.212846 23622 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:13:11.213244 23622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: Bootstrap starting.
I20260620 23:13:11.215543 23622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:13:11.215763 23622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: Bootstrap complete.
I20260620 23:13:11.215955 23622 raft_consensus.cc:348] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } }
I20260620 23:13:11.216053 23622 raft_consensus.cc:729] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f235a868911c4de5b05381f909d97a75, State: Initialized, Role: LEARNER
I20260620 23:13:11.216096 23622 consensus_queue.cc:260] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } }
I20260620 23:13:11.216293 23622 sys_catalog.cc:564] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:11.216267 23625 sys_catalog.cc:455] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.216460 23625 sys_catalog.cc:458] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:13:11.218580 23636 catalog_manager.cc:1259] Loaded cluster ID: d50a1abbe97d4865bcba641d0523acfe
I20260620 23:13:11.218644 23636 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: loading cluster ID for follower catalog manager: success
I20260620 23:13:11.219324 23636 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: acquiring CA information for follower catalog manager: success
I20260620 23:13:11.219576 23636 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 23:13:11.273155 23546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:11.273416 23546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:11.273450 23546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:11.275029 23546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:11.275089 23546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:11.275127 23546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:11.277173 23546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab
--principal=oryx/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:46527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=45637
--tserver_master_addrs=127.16.79.62:41873,127.16.79.61:39717,127.16.79.60:46161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:11.277408 23546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:11.277655 23546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:11.283207 23644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:11.283291 23641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:11.283229 23642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:11.283723 23546 server_base.cc:1034] running on GCE node
I20260620 23:13:11.283902 23546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:11.284125 23546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:11.285280 23546 hybrid_clock.cc:648] HybridClock initialized: now 1781997191285266 us; error 25 us; skew 500 ppm
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:11.288035 23546 init.cc:377] Logged in from keytab as oryx/127.16.79.1@KRBTEST.COM (short username oryx)
I20260620 23:13:11.288390 23546 webserver.cc:460] Webserver started at http://127.16.79.1:45637/ using document root <none> and password file <none>
I20260620 23:13:11.288625 23546 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:11.288731 23546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:11.290072 23546 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:11.290683 23651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.290854 23546 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:11.290943 23546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "58731d03947949188070b9cc06fd12a4"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "cd83ca63348595100a9b3f6f427f4f67"
server_key_iv: "385a25b2f5d2956715fc03b31cac99c6"
server_key_version: "encryptionkey@0"
I20260620 23:13:11.291581 23546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:11.306965 23546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:11.309190 23546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:11.309355 23546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:11.309573 23546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:11.309864 23546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:11.309916 23546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.309955 23546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:11.309983 23546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.316519 23546 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:46527
I20260620 23:13:11.316594 23764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:46527 every 8 connection(s)
I20260620 23:13:11.316943 23546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:11.317978 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 23546
I20260620 23:13:11.318131 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 22998
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.61@KRBTEST.COM
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.60@KRBTEST.COM
I20260620 23:13:11.325779 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:40199
--local_ip_for_outbound_sockets=127.16.79.2
--tserver_master_addrs=127.16.79.62:41873
--webserver_port=44209
--webserver_interface=127.16.79.2
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab
--principal=oryx/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.79.62:41873,127.16.79.61:39717
--tserver_master_addrs=127.16.79.62:41873,127.16.79.61:39717,127.16.79.60:46161 with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:13:11.327569 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.317645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:58049 (local address 127.16.79.61:39717)
0620 23:13:11.317692 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.317694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.317982 (+ 288us) server_negotiation.cc:407] Connection header received
0620 23:13:11.318709 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.318710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.318722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.318787 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.319676 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.320156 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.321219 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.321416 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.321502 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.324229 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.324231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.324231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.324238 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.326057 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.326462 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.326463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.326464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.326517 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.326958 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.326959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.326960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.327079 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.327155 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.327402 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.327476 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:13:11.327767 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.317668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39717 (local address 127.16.79.1:58049)
0620 23:13:11.317864 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.317903 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.318609 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.318852 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.318856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.319394 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.319576 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.319580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.320280 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.320281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.321121 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.321123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.321510 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.321510 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.321627 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.322224 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.322237 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.324130 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.326140 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.326146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.326149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.326370 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.326592 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.326593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.326593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.326877 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.327147 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.327153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.327279 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.327384 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.327534 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260620 23:13:11.328277 23767 heartbeater.cc:347] Connected to a master server at 127.16.79.61:39717
I20260620 23:13:11.328364 23767 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:11.328537 23767 heartbeater.cc:511] Master 127.16.79.61:39717 requested a full tablet report, sending...
I20260620 23:13:11.328663 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.317668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46161 (local address 127.16.79.1:51313)
0620 23:13:11.318097 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.318109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.318620 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.318942 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.318943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.319391 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.319578 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.319583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.320903 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.320904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.322200 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.322203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.322768 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.322770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.322847 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.323288 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.323294 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.324905 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.327572 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.327573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.327574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.327768 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.327950 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.327951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.327952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.328004 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.328314 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.328315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.328391 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.328461 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.328556 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":4,"spinlock_wait_cycles":384,"thread_start_us":116,"threads_started":1}
I20260620 23:13:11.328982 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.318164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:51313 (local address 127.16.79.60:46161)
0620 23:13:11.318374 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.318376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.318385 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:11.318837 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.318838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.318849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.318922 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.319699 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.320816 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.322301 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.322657 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.322717 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.324990 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.324992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.324992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.324999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.327488 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.327839 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.327840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.327841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.327891 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.328073 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.328074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.328074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.328221 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.328290 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.328485 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.328560 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":51,"threads_started":1}
I20260620 23:13:11.329056 23399 ts_manager.cc:194] Registered new tserver with Master: 58731d03947949188070b9cc06fd12a4 (127.16.79.1:46527)
I20260620 23:13:11.329456 23765 heartbeater.cc:347] Connected to a master server at 127.16.79.60:46161
I20260620 23:13:11.329507 23765 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:11.329573 23765 heartbeater.cc:511] Master 127.16.79.60:46161 requested a full tablet report, sending...
I20260620 23:13:11.329914 23586 ts_manager.cc:194] Registered new tserver with Master: 58731d03947949188070b9cc06fd12a4 (127.16.79.1:46527)
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
I20260620 23:13:11.333037 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.318261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:44349 (local address 127.16.79.62:41873)
0620 23:13:11.318306 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.318307 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.324934 (+ 6627us) server_negotiation.cc:407] Connection header received
0620 23:13:11.325034 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.325035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.325047 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.325076 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.325442 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.326150 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.328106 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.328424 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.328489 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.330536 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.330537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.330538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.330543 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.331953 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.332355 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.332356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.332356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.332401 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.332591 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.332592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.332592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.332681 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.332781 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.332927 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.332980 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:13:11.333379 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.317955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.1:44349)
0620 23:13:11.324867 (+ 6912us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.324878 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.324959 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.325139 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.325140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.325292 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.325349 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.325351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.327157 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.327158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.328006 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.328008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.328539 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.328540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.328606 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.329015 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.329020 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.330441 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.332034 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.332036 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.332036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.332279 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.332464 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.332464 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.332465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.332531 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.332778 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.332779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.332826 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.332913 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.333020 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6894,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260620 23:13:11.333833 23766 heartbeater.cc:347] Connected to a master server at 127.16.79.62:41873
I20260620 23:13:11.333889 23766 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:11.333956 23766 heartbeater.cc:511] Master 127.16.79.62:41873 requested a full tablet report, sending...
I20260620 23:13:11.334189 22802 ts_manager.cc:194] Re-registered known tserver with Master: 58731d03947949188070b9cc06fd12a4 (127.16.79.1:46527)
I20260620 23:13:11.334569 22802 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:44349
W20260620 23:13:11.411228 23773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:11.411443 23773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:11.411481 23773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:11.412599 23773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:11.412667 23773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:11.412726 23773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:11.414150 23773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab
--principal=oryx/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:40199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=44209
--tserver_master_addrs=127.16.79.62:41873,127.16.79.61:39717,127.16.79.60:46161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:11.414409 23773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:11.414625 23773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:11.419726 23781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:11.419726 23778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:11.419792 23779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:11.420001 23773 server_base.cc:1034] running on GCE node
I20260620 23:13:11.420228 23773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:11.420408 23773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:11.421545 23773 hybrid_clock.cc:648] HybridClock initialized: now 1781997191421533 us; error 26 us; skew 500 ppm
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:11.423920 23773 init.cc:377] Logged in from keytab as oryx/127.16.79.2@KRBTEST.COM (short username oryx)
I20260620 23:13:11.424361 23773 webserver.cc:460] Webserver started at http://127.16.79.2:44209/ using document root <none> and password file <none>
I20260620 23:13:11.424582 23773 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:11.424662 23773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:11.425935 23773 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:11.426543 23788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.426729 23773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:11.426831 23773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "edb8acd7a81a4495889f5a4c3c83a47d"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "7cc6a21ab5061c6f4f2cad49bb1449c1"
server_key_iv: "32b79b7a937d34bae92bb46e12ee2e4c"
server_key_version: "encryptionkey@0"
I20260620 23:13:11.427126 23773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:11.439157 23773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:11.441288 23773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:11.441433 23773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:11.441632 23773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:11.441903 23773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:11.441955 23773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.441993 23773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:11.442020 23773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.448319 23773 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:40199
I20260620 23:13:11.448362 23901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:40199 every 8 connection(s)
I20260620 23:13:11.448792 23773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:13:11.453226 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 23773
I20260620 23:13:11.453398 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 23132
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.60@KRBTEST.COM
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.61@KRBTEST.COM
I20260620 23:13:11.460362 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.449864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:51357 (local address 127.16.79.60:46161)
0620 23:13:11.449913 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.449915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.450021 (+ 106us) server_negotiation.cc:407] Connection header received
0620 23:13:11.450769 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.450770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.450784 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.450828 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.451561 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.452422 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.453709 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.454097 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.454142 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.456752 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.456753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.456754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.456761 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.458643 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.459177 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.459178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.459179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.459236 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.459554 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.459555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.459555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.459708 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.459755 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.460110 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.460205 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:13:11.460902 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.449622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:50989 (local address 127.16.79.62:41873)
0620 23:13:11.449683 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.449685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.449766 (+ 81us) server_negotiation.cc:407] Connection header received
0620 23:13:11.450744 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.450745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.450759 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.450800 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.451537 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.452041 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.453189 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.453474 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.453514 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.456965 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.456966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.456966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.456973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.459049 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.459770 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.459772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.459772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.459845 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.460229 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.460230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.460230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.460396 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.460445 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.460672 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.460779 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 23:13:11.460969 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.449785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46161 (local address 127.16.79.2:51357)
0620 23:13:11.449933 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.449943 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.450658 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.450932 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.450934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.451294 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.451449 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.451454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.452520 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.452521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.453604 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.453607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.454232 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.454233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.454319 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.454720 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.454726 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.456615 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.458733 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.458739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.458741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.459085 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.459326 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.459328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.459328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.459467 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.459805 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.459810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.459965 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.460485 (+ 520us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.460668 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":58,"thread_start_us":54,"threads_started":1}
I20260620 23:13:11.461215 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.449502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.2:50989)
0620 23:13:11.449658 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.449693 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.450604 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.450931 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.450934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.451273 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.451447 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.451451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.452143 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.452143 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.453091 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.453093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.453597 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.453598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.453739 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.454248 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.454258 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.456873 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.459199 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.459201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.459202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.459649 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.459927 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.459928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.459928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.460112 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.460499 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.460500 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.460542 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.460973 (+ 431us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.461071 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260620 23:13:11.461387 23902 heartbeater.cc:347] Connected to a master server at 127.16.79.60:46161
I20260620 23:13:11.461483 23902 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:11.461658 23902 heartbeater.cc:511] Master 127.16.79.60:46161 requested a full tablet report, sending...
I20260620 23:13:11.461665 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:44657
--local_ip_for_outbound_sockets=127.16.79.3
--tserver_master_addrs=127.16.79.62:41873
--webserver_port=34345
--webserver_interface=127.16.79.3
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab
--principal=oryx/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.79.62:41873,127.16.79.61:39717
--tserver_master_addrs=127.16.79.62:41873,127.16.79.61:39717,127.16.79.60:46161 with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:13:11.462116 23586 ts_manager.cc:194] Registered new tserver with Master: edb8acd7a81a4495889f5a4c3c83a47d (127.16.79.2:40199)
I20260620 23:13:11.463503 23903 heartbeater.cc:347] Connected to a master server at 127.16.79.62:41873
I20260620 23:13:11.463575 23903 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:11.463613 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.449498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:54283 (local address 127.16.79.61:39717)
0620 23:13:11.449553 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.449555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.449803 (+ 248us) server_negotiation.cc:407] Connection header received
0620 23:13:11.450732 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.450733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.450746 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.450784 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.451560 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.452407 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.453617 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.454199 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.454243 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.456964 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.456966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.456966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.457540 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.460730 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.461860 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.461861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.461862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.461921 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.463069 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.463070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.463070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.463214 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.463297 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.463446 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.463515 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:13:11.463641 23903 heartbeater.cc:511] Master 127.16.79.62:41873 requested a full tablet report, sending...
I20260620 23:13:11.463877 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.449503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39717 (local address 127.16.79.2:54283)
0620 23:13:11.449731 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.449743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.450623 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.450932 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.450934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.451299 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.451449 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.451454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.452520 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.452521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.453529 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.453531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.454331 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.454332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.454399 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.455024 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.455029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.456864 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.461585 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.461586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.461587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.461797 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.461997 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.461998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.461998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.462053 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.463284 (+ 1231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.463285 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.463335 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.463424 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.463514 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":87,"thread_start_us":97,"threads_started":1}
I20260620 23:13:11.463900 22802 ts_manager.cc:194] Re-registered known tserver with Master: edb8acd7a81a4495889f5a4c3c83a47d (127.16.79.2:40199)
I20260620 23:13:11.464170 23904 heartbeater.cc:347] Connected to a master server at 127.16.79.61:39717
I20260620 23:13:11.464216 23904 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:11.464273 23904 heartbeater.cc:511] Master 127.16.79.61:39717 requested a full tablet report, sending...
I20260620 23:13:11.464385 22802 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:50989
I20260620 23:13:11.464517 23399 ts_manager.cc:194] Registered new tserver with Master: edb8acd7a81a4495889f5a4c3c83a47d (127.16.79.2:40199)
W20260620 23:13:11.542001 23909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:11.542222 23909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:11.542261 23909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:11.543432 23909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:11.543495 23909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:11.543545 23909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:11.545100 23909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab
--principal=oryx/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:44657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=34345
--tserver_master_addrs=127.16.79.62:41873,127.16.79.61:39717,127.16.79.60:46161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:11.545351 23909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:11.545564 23909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:11.550592 23914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:11.550602 23915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:11.550602 23917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:11.550853 23909 server_base.cc:1034] running on GCE node
I20260620 23:13:11.551047 23909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:11.551317 23909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:11.552474 23909 hybrid_clock.cc:648] HybridClock initialized: now 1781997191552454 us; error 42 us; skew 500 ppm
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:11.555117 23909 init.cc:377] Logged in from keytab as oryx/127.16.79.3@KRBTEST.COM (short username oryx)
I20260620 23:13:11.555606 23909 webserver.cc:460] Webserver started at http://127.16.79.3:34345/ using document root <none> and password file <none>
I20260620 23:13:11.555838 23909 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:11.555903 23909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:11.557375 23909 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:13:11.557981 23924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.558300 23909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:11.558382 23909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9"
format_stamp: "Formatted at 2026-06-20 23:13:10 on dist-test-slave-pswb"
server_key: "886b0c79c739f865b40932b9284a09dd"
server_key_iv: "f620b4f65d4e29232e5b8b38429d181b"
server_key_version: "encryptionkey@0"
I20260620 23:13:11.558653 23909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:11.574564 23909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:11.576869 23909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:11.576977 23909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:11.577206 23909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:11.577502 23909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:11.577534 23909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.577556 23909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:11.577602 23909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:11.583189 23909 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:44657
I20260620 23:13:11.583315 24037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:44657 every 8 connection(s)
I20260620 23:13:11.583636 23909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:13:11.587278 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 23909
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.60@KRBTEST.COM
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
Jun 20 23:13:11 dist-test-slave-pswb krb5kdc[22757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.61@KRBTEST.COM
I20260620 23:13:11.598593 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.584933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:52297 (local address 127.16.79.60:46161)
0620 23:13:11.584989 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.584991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.585104 (+ 113us) server_negotiation.cc:407] Connection header received
0620 23:13:11.585764 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.585766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.585778 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.585851 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.586542 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.587281 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.588864 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.589157 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.589197 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.594839 (+ 5642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.594840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.594840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.594848 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.596794 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.597292 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.597294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.597294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.597347 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.597780 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.597781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.597782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.597930 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.598001 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.598409 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.598492 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:13:11.599505 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.589147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.16.79.60:46161)
0620 23:13:11.589400 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.589402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.590588 (+ 1186us) server_negotiation.cc:407] Connection header received
0620 23:13:11.590610 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.590611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.590624 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.590684 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.591498 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.592265 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.593325 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.593662 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.593726 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.595676 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.595678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.595678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.595689 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.598023 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.598464 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.598465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.598465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.598520 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.598721 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.598722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.598722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.598883 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.598979 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.599318 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.599393 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":44,"threads_started":1}
I20260620 23:13:11.599864 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.584855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46161 (local address 127.16.79.3:52297)
0620 23:13:11.585027 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.585035 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.585404 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.585861 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.585861 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.586197 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.586438 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.586443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.587416 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.587417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.588762 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.588764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.589347 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.589347 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.589495 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.590395 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.590401 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.594752 (+ 4351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.596885 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.596892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.596896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.597209 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.597423 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.597423 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.597424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.597559 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.598043 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.598050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.598214 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.599064 (+ 850us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.599388 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":66,"thread_start_us":70,"threads_started":1}
I20260620 23:13:11.600354 24038 heartbeater.cc:347] Connected to a master server at 127.16.79.60:46161
I20260620 23:13:11.600451 24038 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:11.600533 24050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.588600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.16.79.61:39717)
0620 23:13:11.589877 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.589879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.589886 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:11.589904 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.589904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.589915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.589994 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.590683 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.591404 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.592648 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.592953 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.593015 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.595284 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.595285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.595285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.595292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.598366 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.598733 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.598734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.598735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.598800 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.598977 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.598977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.598978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.599089 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.599188 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.599549 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.600442 (+ 893us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259,"thread_start_us":63,"threads_started":1}
I20260620 23:13:11.600641 24038 heartbeater.cc:511] Master 127.16.79.60:46161 requested a full tablet report, sending...
I20260620 23:13:11.601022 23586 ts_manager.cc:194] Registered new tserver with Master: 5eb4a8ac81a14379b3ba1bec6b8e25b9 (127.16.79.3:44657)
I20260620 23:13:11.601200 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.584571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:37907 (local address 127.16.79.62:41873)
0620 23:13:11.584666 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.584669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.584865 (+ 196us) server_negotiation.cc:407] Connection header received
0620 23:13:11.585577 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.585578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.585593 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.585681 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.586541 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.587287 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.588864 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.589157 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.589197 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.595963 (+ 6766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.595964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.595964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.595971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.598912 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.600082 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.600083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.600083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.600137 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.600335 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.600336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.600336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.600469 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.600520 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.601014 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.601096 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:13:11.601593 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.584456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39717 (local address 127.16.79.3:58217)
0620 23:13:11.584781 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.584794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.585412 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.586138 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.586138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.586284 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.586437 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.586441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.588235 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.588236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.589085 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.589088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.589623 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.589623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.589696 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.590248 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.590257 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.593160 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.599855 (+ 6695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.599856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.599856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.600388 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.600902 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.600903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.600903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.600964 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.601232 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.601233 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.601274 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.601383 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.601478 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":2,"thread_start_us":155,"threads_started":1}
I20260620 23:13:11.601624 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.584468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:58217 (local address 127.16.79.61:39717)
0620 23:13:11.584601 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.584604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.584887 (+ 283us) server_negotiation.cc:407] Connection header received
0620 23:13:11.585676 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.585677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.585692 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.585750 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.586536 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.587027 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.589193 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.589501 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.589542 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.593267 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.593630 (+ 363us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.593631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.593639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.599765 (+ 6126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.600772 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.600773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.600774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.600835 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.601035 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.601036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.601036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.601151 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.601199 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.601440 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.601516 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:13:11.601929 24040 heartbeater.cc:347] Connected to a master server at 127.16.79.61:39717
I20260620 23:13:11.602079 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.584455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.3:37907)
0620 23:13:11.584729 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.584772 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.585399 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.585763 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.585767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.586184 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:11.586438 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.586443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.587416 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.587417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.588762 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.588764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.589344 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.589345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.589495 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.590515 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:11.590520 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:11.595797 (+ 5277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:11.599784 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.599785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.599785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.599994 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.600198 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:11.600199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:11.600200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:11.600257 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:11.600836 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:11.600837 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:11.600887 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.601459 (+ 572us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.601536 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":69,"spinlock_wait_cycles":7936,"thread_start_us":121,"threads_started":1}
I20260620 23:13:11.602165 24040 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:11.602244 24040 heartbeater.cc:511] Master 127.16.79.61:39717 requested a full tablet report, sending...
I20260620 23:13:11.602488 23399 ts_manager.cc:194] Registered new tserver with Master: 5eb4a8ac81a14379b3ba1bec6b8e25b9 (127.16.79.3:44657)
I20260620 23:13:11.602643 24039 heartbeater.cc:347] Connected to a master server at 127.16.79.62:41873
I20260620 23:13:11.602700 24039 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:11.602764 24039 heartbeater.cc:511] Master 127.16.79.62:41873 requested a full tablet report, sending...
I20260620 23:13:11.602900 24051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.588827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.16.79.62:41873)
0620 23:13:11.589596 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.589598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.589755 (+ 157us) server_negotiation.cc:407] Connection header received
0620 23:13:11.589777 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.589778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.589791 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.589897 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.591742 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.592565 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.593884 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.594205 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.594272 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.597208 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.597210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.597210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.597217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.601668 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.602038 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.602039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.602040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.602103 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.602344 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.602345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.602345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.602466 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.602560 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.602697 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.602785 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":65,"threads_started":1}
I20260620 23:13:11.602962 22802 ts_manager.cc:194] Re-registered known tserver with Master: 5eb4a8ac81a14379b3ba1bec6b8e25b9 (127.16.79.3:44657)
I20260620 23:13:11.603279 22802 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:37907
I20260620 23:13:11.605180 22802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:13:11.606680 22802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:13:11.613790 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.610304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:46527 (local address 127.0.0.1:37322)
0620 23:13:11.610497 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.610508 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.610548 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.610732 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.610733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.610735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:11.610775 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.610776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.611813 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.611823 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.612640 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.612642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.613558 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.613559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.613617 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.613621 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.613676 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.613722 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":3}
I20260620 23:13:11.613801 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.610402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.16.79.1:46527)
0620 23:13:11.610553 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.610558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.610592 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:13:11.610632 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.610634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.610647 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.610734 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:11.610829 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.611720 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.612775 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.613462 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.613550 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.613622 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.613690 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.613727 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260620 23:13:11.614401 23699 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca0a2b8a9ec4652bde60bee194aea7c (DEFAULT_TABLE table=test-table [id=f1180669016b451684fe6cd3d1549cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:11.614699 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.610304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:44657 (local address 127.0.0.1:60172)
0620 23:13:11.610598 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.610608 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.610648 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.610926 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.610926 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.610928 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:11.610956 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.610958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.612145 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.612146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.613236 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.613239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.614446 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.614447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.614515 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.614519 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.614567 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.614609 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":66,"thread_start_us":79,"threads_started":1}
I20260620 23:13:11.614758 23699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca0a2b8a9ec4652bde60bee194aea7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:11.614929 24059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.610401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:40199 (local address 127.0.0.1:36792)
0620 23:13:11.610784 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.610794 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.610835 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.611140 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.611142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.611144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:11.611182 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.611183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.612364 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.612365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.613476 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.613478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.614552 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.614554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.614647 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.614651 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.614729 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.614865 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":6,"thread_start_us":60,"threads_started":1}
I20260620 23:13:11.614976 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.610461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.16.79.3:44657)
0620 23:13:11.610764 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.610768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.610778 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:13:11.610814 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.610816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.610831 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.610927 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:11.611008 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.612055 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.613318 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.614017 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.614081 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.614405 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.614870 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.614909 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":66,"threads_started":1}
I20260620 23:13:11.614959 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.610631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.16.79.2:40199)
0620 23:13:11.610938 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.610942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.610950 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:13:11.610986 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.610988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.611003 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.611072 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:11.611268 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.612284 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.613566 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.614450 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.614531 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.614639 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.614838 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.614886 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":75,"threads_started":1}
I20260620 23:13:11.615549 23972 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca0a2b8a9ec4652bde60bee194aea7c (DEFAULT_TABLE table=test-table [id=f1180669016b451684fe6cd3d1549cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:11.615898 23972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca0a2b8a9ec4652bde60bee194aea7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:11.617743 24064 tablet_bootstrap.cc:492] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4: Bootstrap starting.
I20260620 23:13:11.618369 24064 tablet_bootstrap.cc:654] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:11.618469 23836 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca0a2b8a9ec4652bde60bee194aea7c (DEFAULT_TABLE table=test-table [id=f1180669016b451684fe6cd3d1549cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:11.618685 24064 log.cc:824] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:11.618790 23836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca0a2b8a9ec4652bde60bee194aea7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:11.619415 24064 tablet_bootstrap.cc:492] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4: No bootstrap required, opened a new log
I20260620 23:13:11.619484 24064 ts_tablet_manager.cc:1397] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:13:11.620913 24065 tablet_bootstrap.cc:492] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Bootstrap starting.
I20260620 23:13:11.621471 24065 tablet_bootstrap.cc:654] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:11.621773 24065 log.cc:824] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:11.622133 24064 raft_consensus.cc:348] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:11.622298 24064 raft_consensus.cc:374] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:11.622332 24064 raft_consensus.cc:729] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58731d03947949188070b9cc06fd12a4, State: Initialized, Role: FOLLOWER
I20260620 23:13:11.622470 24064 consensus_queue.cc:260] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:11.622743 24064 ts_tablet_manager.cc:1428] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 23:13:11.622792 23766 heartbeater.cc:503] Master 127.16.79.62:41873 was elected leader, sending a full tablet report...
I20260620 23:13:11.622937 24067 tablet_bootstrap.cc:492] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d: Bootstrap starting.
I20260620 23:13:11.623484 24067 tablet_bootstrap.cc:654] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:11.623797 24067 log.cc:824] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:11.624516 24065 tablet_bootstrap.cc:492] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9: No bootstrap required, opened a new log
I20260620 23:13:11.624528 24067 tablet_bootstrap.cc:492] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d: No bootstrap required, opened a new log
I20260620 23:13:11.624588 24067 ts_tablet_manager.cc:1397] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:13:11.624588 24065 ts_tablet_manager.cc:1397] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 23:13:11.626572 24065 raft_consensus.cc:348] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:11.626726 24065 raft_consensus.cc:374] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:11.626758 24065 raft_consensus.cc:729] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb4a8ac81a14379b3ba1bec6b8e25b9, State: Initialized, Role: FOLLOWER
I20260620 23:13:11.626856 24065 consensus_queue.cc:260] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:11.626971 24067 raft_consensus.cc:348] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:11.627105 24067 raft_consensus.cc:374] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:11.627113 24065 ts_tablet_manager.cc:1428] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:13:11.627153 24067 raft_consensus.cc:729] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb8acd7a81a4495889f5a4c3c83a47d, State: Initialized, Role: FOLLOWER
I20260620 23:13:11.627272 24067 consensus_queue.cc:260] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:11.627277 24039 heartbeater.cc:503] Master 127.16.79.62:41873 was elected leader, sending a full tablet report...
I20260620 23:13:11.627503 24067 ts_tablet_manager.cc:1428] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:13:11.627903 23903 heartbeater.cc:503] Master 127.16.79.62:41873 was elected leader, sending a full tablet report...
I20260620 23:13:11.656059 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.653665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.16.79.60:46161)
0620 23:13:11.653728 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.653730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.653737 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:11.653758 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.653759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.653777 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.653844 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:11.653963 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.654430 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.655345 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.655794 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.655852 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.655897 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.655978 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.656011 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:13:11.656049 24059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.653580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46161 (local address 127.0.0.1:47390)
0620 23:13:11.653632 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.653650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.653685 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.653862 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.653863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.653864 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:11.653903 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.653904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.654491 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.654491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.655244 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.655246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.655866 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.655867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.655911 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.655914 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.655960 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.655995 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 23:13:11.656442 23596 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260620 23:13:11.657688 23625 sys_catalog.cc:455] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36ed237bf6274042870df583d008b757. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.657812 23625 sys_catalog.cc:458] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: This master's current role is: LEARNER
W20260620 23:13:11.699613 23905 tablet.cc:2369] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:13:11.740942 24072 raft_consensus.cc:482] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:11.741046 24072 raft_consensus.cc:504] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:11.741385 24072 leader_election.cc:288] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5eb4a8ac81a14379b3ba1bec6b8e25b9 (127.16.79.3:44657), 58731d03947949188070b9cc06fd12a4 (127.16.79.1:46527)
I20260620 23:13:11.745308 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.741636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:53467 (local address 127.16.79.3:44657)
0620 23:13:11.741683 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.741685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.741695 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:13:11.741726 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.741727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.741737 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.741802 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:11.742018 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.742769 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.744327 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.744903 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.744980 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.744989 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.745192 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.745238 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:13:11.745308 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.741657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:44871 (local address 127.16.79.1:46527)
0620 23:13:11.741699 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.741702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.741709 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:11.741732 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.741732 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.741743 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.741807 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:11.742018 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.742773 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.744339 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.744903 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.744980 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.744989 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.745192 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.745237 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:13:11.745311 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.741574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:46527 (local address 127.16.79.2:44871)
0620 23:13:11.741631 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.741641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.741674 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.741822 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.741823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.741824 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:11.741915 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.741920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.742865 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.742867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.744169 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.744172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.744990 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.744991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.745088 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.745092 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.745156 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.745207 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":114}
I20260620 23:13:11.745391 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.741509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:44657 (local address 127.16.79.2:53467)
0620 23:13:11.741558 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:11.741570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:11.741636 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:11.741815 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:11.741816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:11.741818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:11.741915 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.741916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.742868 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.742869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.744169 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:11.744172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.744990 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.744991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:11.745088 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.745092 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:11.745156 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:13:11.745207 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":86}
I20260620 23:13:11.745699 23719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca0a2b8a9ec4652bde60bee194aea7c" candidate_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58731d03947949188070b9cc06fd12a4" is_pre_election: true
I20260620 23:13:11.745888 23719 raft_consensus.cc:2454] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edb8acd7a81a4495889f5a4c3c83a47d in term 0.
I20260620 23:13:11.745995 23992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca0a2b8a9ec4652bde60bee194aea7c" candidate_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" is_pre_election: true
I20260620 23:13:11.746145 23992 raft_consensus.cc:2454] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edb8acd7a81a4495889f5a4c3c83a47d in term 0.
I20260620 23:13:11.746125 23791 leader_election.cc:302] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58731d03947949188070b9cc06fd12a4, edb8acd7a81a4495889f5a4c3c83a47d; no voters:
I20260620 23:13:11.746268 24072 raft_consensus.cc:2781] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:13:11.746330 24072 raft_consensus.cc:482] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:11.746364 24072 raft_consensus.cc:3037] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:11.747148 24072 raft_consensus.cc:504] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:11.747290 24072 leader_election.cc:288] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [CANDIDATE]: Term 1 election: Requested vote from peers 5eb4a8ac81a14379b3ba1bec6b8e25b9 (127.16.79.3:44657), 58731d03947949188070b9cc06fd12a4 (127.16.79.1:46527)
I20260620 23:13:11.747449 23719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca0a2b8a9ec4652bde60bee194aea7c" candidate_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58731d03947949188070b9cc06fd12a4"
I20260620 23:13:11.747443 23992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca0a2b8a9ec4652bde60bee194aea7c" candidate_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9"
I20260620 23:13:11.747522 23719 raft_consensus.cc:3037] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:11.747524 23992 raft_consensus.cc:3037] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:11.748148 23719 raft_consensus.cc:2454] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edb8acd7a81a4495889f5a4c3c83a47d in term 1.
I20260620 23:13:11.748281 23992 raft_consensus.cc:2454] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edb8acd7a81a4495889f5a4c3c83a47d in term 1.
I20260620 23:13:11.748368 23791 leader_election.cc:302] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58731d03947949188070b9cc06fd12a4, edb8acd7a81a4495889f5a4c3c83a47d; no voters:
I20260620 23:13:11.748456 24072 raft_consensus.cc:2781] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:11.748620 24072 raft_consensus.cc:686] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 1 LEADER]: Becoming Leader. State: Replica: edb8acd7a81a4495889f5a4c3c83a47d, State: Running, Role: LEADER
I20260620 23:13:11.748761 24072 consensus_queue.cc:237] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:11.749585 22802 catalog_manager.cc:5495] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d reported cstate change: term changed from 0 to 1, leader changed from <none> to edb8acd7a81a4495889f5a4c3c83a47d (127.16.79.2). New cstate: current_term: 1 leader_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } health_report { overall_health: HEALTHY } } }
I20260620 23:13:11.750161 23549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: NON_VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: true } } }
I20260620 23:13:11.750248 23549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:11.750725 23549 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757: attempting to promote NON_VOTER f235a868911c4de5b05381f909d97a75 to VOTER
I20260620 23:13:11.750866 23549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } }
I20260620 23:13:11.751370 23596 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 LEARNER]: Refusing update from remote peer 36ed237bf6274042870df583d008b757: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 23:13:11.751457 23409 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 FOLLOWER]: Refusing update from remote peer 36ed237bf6274042870df583d008b757: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 23:13:11.751562 24078 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260620 23:13:11.751647 24081 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 23:13:11.752300 24081 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f235a868911c4de5b05381f909d97a75 (127.16.79.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } } }
I20260620 23:13:11.752420 23409 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f235a868911c4de5b05381f909d97a75 (127.16.79.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } } }
I20260620 23:13:11.752476 23596 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f235a868911c4de5b05381f909d97a75 (127.16.79.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } } }
I20260620 23:13:11.753260 24081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } } }
I20260620 23:13:11.753340 24081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:11.753325 23625 sys_catalog.cc:455] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } } }
I20260620 23:13:11.753404 23625 sys_catalog.cc:458] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:13:11.753394 24081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } } }
I20260620 23:13:11.753459 24081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:11.753535 23549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } } }
I20260620 23:13:11.753612 23549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:11.753552 24082 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36ed237bf6274042870df583d008b757" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } } }
I20260620 23:13:11.753731 24082 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:13:11.757612 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.755240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.16.79.2:40199)
0620 23:13:11.755302 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.755304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.755312 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:13:11.755340 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.755341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.755357 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.755435 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:13:11.755557 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.756066 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.756895 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.757080 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.757154 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.757312 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:13:11.757401 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:13:11.757481 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.757522 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.757541 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:13:11.759114 23992 raft_consensus.cc:1261] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 1 FOLLOWER]: Refusing update from remote peer edb8acd7a81a4495889f5a4c3c83a47d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:11.759133 23719 raft_consensus.cc:1261] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4 [term 1 FOLLOWER]: Refusing update from remote peer edb8acd7a81a4495889f5a4c3c83a47d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:11.759433 24072 consensus_queue.cc:1035] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:11.759580 24076 consensus_queue.cc:1035] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:11.762002 24084 mvcc.cc:205] Tried to move back new op lower bound from 7299060497443123200 to 7299060497403506688. Current Snapshot: MvccSnapshot[applied={T|T < 7299060497443123200}]
I20260620 23:13:11.764016 22801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:11.764099 22801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:11.764988 24085 mvcc.cc:205] Tried to move back new op lower bound from 7299060497443123200 to 7299060497403506688. Current Snapshot: MvccSnapshot[applied={T|T < 7299060497443123200}]
I20260620 23:13:11.765769 22801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36ed237bf6274042870df583d008b757: Sending DeleteTablet for 3 replicas of tablet 8ca0a2b8a9ec4652bde60bee194aea7c
I20260620 23:13:11.766058 23972 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca0a2b8a9ec4652bde60bee194aea7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:11 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:60172
I20260620 23:13:11.766083 23699 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca0a2b8a9ec4652bde60bee194aea7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:11 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:37322
I20260620 23:13:11.766108 23836 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca0a2b8a9ec4652bde60bee194aea7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:11 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:36792
I20260620 23:13:11.766258 23835 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca0a2b8a9ec4652bde60bee194aea7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:11 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:36792
I20260620 23:13:11.766799 24093 tablet_replica.cc:330] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4: stopping tablet replica
I20260620 23:13:11.766851 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 22770
I20260620 23:13:11.766904 24093 raft_consensus.cc:2229] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:11.766927 24095 tablet_replica.cc:330] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d: stopping tablet replica
I20260620 23:13:11.766995 24095 raft_consensus.cc:2229] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:13:11.767010 24093 raft_consensus.cc:2258] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:11.767074 24095 raft_consensus.cc:2258] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:11.767495 24095 ts_tablet_manager.cc:1905] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:11.768527 24093 ts_tablet_manager.cc:1905] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:11.769198 24095 ts_tablet_manager.cc:1918] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:11.769281 24095 log.cc:1196] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/8ca0a2b8a9ec4652bde60bee194aea7c
I20260620 23:13:11.769570 24095 ts_tablet_manager.cc:1939] T 8ca0a2b8a9ec4652bde60bee194aea7c P edb8acd7a81a4495889f5a4c3c83a47d: Deleting consensus metadata
I20260620 23:13:11.770087 24093 ts_tablet_manager.cc:1918] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:11.770141 24093 log.cc:1196] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-0/wal/wals/8ca0a2b8a9ec4652bde60bee194aea7c
I20260620 23:13:11.770316 24093 ts_tablet_manager.cc:1939] T 8ca0a2b8a9ec4652bde60bee194aea7c P 58731d03947949188070b9cc06fd12a4: Deleting consensus metadata
I20260620 23:13:11.770537 24094 tablet_replica.cc:330] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9: stopping tablet replica
I20260620 23:13:11.770630 24094 raft_consensus.cc:2229] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:11.770725 24094 raft_consensus.cc:2258] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 23:13:11.774612 23653 connection.cc:535] server connection from 127.0.0.1:37322 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 23:13:11.774863 23789 connection.cc:698] server connection from 127.0.0.1:36792 send error: Network error: sendmsg error: Broken pipe (error 32)
W20260620 23:13:11.774924 23789 connection.cc:441] server connection from 127.0.0.1:36792 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36792 (request call id 2) could send its response
I20260620 23:13:11.775326 24094 ts_tablet_manager.cc:1905] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:11.776643 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.766635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.16.79.60:46161)
0620 23:13:11.766673 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.766675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.767316 (+ 641us) server_negotiation.cc:407] Connection header received
0620 23:13:11.767416 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.767416 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.767427 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.767456 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.770473 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.771144 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.771899 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.772112 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.772186 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.773805 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.773807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.773807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.773813 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.775419 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.775796 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.775797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.775797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.775850 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.776053 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.776054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.776054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.776173 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.776266 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.776488 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.776560 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:13:11.777011 24094 ts_tablet_manager.cc:1918] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:11.777100 24094 log.cc:1196] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/8ca0a2b8a9ec4652bde60bee194aea7c
I20260620 23:13:11.777369 24094 ts_tablet_manager.cc:1939] T 8ca0a2b8a9ec4652bde60bee194aea7c P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Deleting consensus metadata
W20260620 23:13:11.777650 23928 connection.cc:441] server connection from 127.0.0.1:60172 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60172 (request call id 1) could send its response
I20260620 23:13:11.784031 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.776793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.16.79.60:46161)
0620 23:13:11.776830 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.776831 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.777140 (+ 309us) server_negotiation.cc:407] Connection header received
0620 23:13:11.777235 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.777236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.777245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.777273 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.777964 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.778556 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.779375 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.779680 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.779747 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.781310 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.781312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.781312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.781318 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.782866 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.783230 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.783231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.783231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.783286 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.783478 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.783479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.783479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.783590 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.783657 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.783857 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.783927 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:13:11.785054 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:11.778127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.16.79.61:39717)
0620 23:13:11.778173 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:11.778175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:11.778244 (+ 69us) server_negotiation.cc:407] Connection header received
0620 23:13:11.778305 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:11.778306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:11.778314 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:11.778372 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:11.778989 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.779625 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.780469 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:11.780673 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:11.780787 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:11.782149 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:11.782151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:11.782151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:11.782158 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:11.783618 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.783984 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.783985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.783986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.784037 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:11.784472 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:11.784473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:11.784473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:11.784613 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:11.784726 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:11.784870 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 23:13:11.784964 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:13:12.763376 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:12.763078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.3:58357)
0620 23:13:12.763242 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:12.763315 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
W20260620 23:13:12.763667 24039 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:41873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
I20260620 23:13:12.766861 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:12.766641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.1:41063)
0620 23:13:12.766756 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:12.766828 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
I20260620 23:13:12.766919 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:12.766686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.2:52977)
0620 23:13:12.766807 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:12.766859 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
W20260620 23:13:12.767030 23766 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:41873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
W20260620 23:13:12.767058 23903 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:41873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
I20260620 23:13:13.266191 24106 raft_consensus.cc:482] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36ed237bf6274042870df583d008b757)
I20260620 23:13:13.266300 24106 raft_consensus.cc:504] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } }
I20260620 23:13:13.266605 24106 leader_election.cc:288] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36ed237bf6274042870df583d008b757 (127.16.79.62:41873), c8d362c900e64b6d829a6ce9d0378a1b (127.16.79.61:39717)
I20260620 23:13:13.266880 24108 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36ed237bf6274042870df583d008b757)
I20260620 23:13:13.267009 24108 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } }
I20260620 23:13:13.267252 24108 leader_election.cc:288] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36ed237bf6274042870df583d008b757 (127.16.79.62:41873), f235a868911c4de5b05381f909d97a75 (127.16.79.60:46161)
I20260620 23:13:13.267257 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.266659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:58956)
0620 23:13:13.266909 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.267184 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":46,"threads_started":1}
W20260620 23:13:13.267426 23572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
I20260620 23:13:13.267585 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.267327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:58960)
0620 23:13:13.267447 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.267530 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":44,"threads_started":1}
W20260620 23:13:13.267793 23385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
I20260620 23:13:13.267889 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.267786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:58972)
0620 23:13:13.267832 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.267858 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 23:13:13.267995 23572 leader_election.cc:334] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36ed237bf6274042870df583d008b757 (127.16.79.62:41873): Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
I20260620 23:13:13.268457 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.268194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:58974)
0620 23:13:13.268395 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.268431 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
W20260620 23:13:13.268651 23385 leader_election.cc:334] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36ed237bf6274042870df583d008b757 (127.16.79.62:41873): Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
I20260620 23:13:13.271065 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.267464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46161 (local address 127.0.0.1:47434)
0620 23:13:13.267672 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.267684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:13.267748 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:13.268365 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:13.268367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:13.268371 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:13.268424 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:13.268425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:13.269295 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:13.269296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:13.270052 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:13.270053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:13.270775 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:13.270776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:13.270845 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:13.270853 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:13:13.270936 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:13:13.270990 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1,"thread_start_us":35,"threads_started":1}
I20260620 23:13:13.271132 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.267634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.16.79.60:46161)
0620 23:13:13.268225 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:13.268227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:13.268234 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:13.268251 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:13.268252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:13.268262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:13.268339 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:13.268527 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:13.269208 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:13.270168 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:13.270661 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:13.270721 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:13.270730 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:13.270973 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 23:13:13.271024 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":562,"thread_start_us":94,"threads_started":1}
I20260620 23:13:13.271255 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.266953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39717 (local address 127.0.0.1:38180)
0620 23:13:13.267257 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.267270 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:13.267539 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:13.268152 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:13.268153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:13.268156 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:13.268236 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:13.268239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:13.269159 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:13.269160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:13.270202 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:13.270204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:13.270977 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:13.270978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:13.271044 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:13.271048 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:13.271117 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 23:13:13.271169 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":61,"threads_started":1}
I20260620 23:13:13.271268 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.267008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.16.79.61:39717)
0620 23:13:13.267187 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:13.267189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:13.268011 (+ 822us) server_negotiation.cc:407] Connection header received
0620 23:13:13.268045 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:13.268046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:13.268065 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:13.268133 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:13.268356 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:13.269070 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:13.270344 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:13.270856 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:13.270943 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:13.270955 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:13.271138 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 23:13:13.271192 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":159,"thread_start_us":51,"threads_started":1}
I20260620 23:13:13.271694 23409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f235a868911c4de5b05381f909d97a75" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" is_pre_election: true
I20260620 23:13:13.271847 23409 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f235a868911c4de5b05381f909d97a75 in term 1.
I20260620 23:13:13.272020 23596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f235a868911c4de5b05381f909d97a75" is_pre_election: true
I20260620 23:13:13.272046 23570 leader_election.cc:302] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c8d362c900e64b6d829a6ce9d0378a1b, f235a868911c4de5b05381f909d97a75; no voters: 36ed237bf6274042870df583d008b757
I20260620 23:13:13.272146 23596 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d362c900e64b6d829a6ce9d0378a1b in term 1.
I20260620 23:13:13.272258 24106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:13:13.272313 24106 raft_consensus.cc:482] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 36ed237bf6274042870df583d008b757)
I20260620 23:13:13.272310 23385 leader_election.cc:302] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c8d362c900e64b6d829a6ce9d0378a1b, f235a868911c4de5b05381f909d97a75; no voters: 36ed237bf6274042870df583d008b757
I20260620 23:13:13.272356 24106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:13:13.272416 24108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:13:13.272459 24108 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 36ed237bf6274042870df583d008b757)
I20260620 23:13:13.272490 24108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:13:13.273533 24106 raft_consensus.cc:504] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } }
I20260620 23:13:13.273604 24108 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } }
I20260620 23:13:13.273767 24106 leader_election.cc:288] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [CANDIDATE]: Term 2 election: Requested vote from peers 36ed237bf6274042870df583d008b757 (127.16.79.62:41873), c8d362c900e64b6d829a6ce9d0378a1b (127.16.79.61:39717)
I20260620 23:13:13.273898 23409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f235a868911c4de5b05381f909d97a75" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8d362c900e64b6d829a6ce9d0378a1b"
I20260620 23:13:13.273980 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.273816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:58996)
0620 23:13:13.273870 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.273906 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 23:13:13.273976 23596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f235a868911c4de5b05381f909d97a75"
I20260620 23:13:13.274045 23409 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f235a868911c4de5b05381f909d97a75 in current term 2: Already voted for candidate c8d362c900e64b6d829a6ce9d0378a1b in this term.
I20260620 23:13:13.274049 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.273775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:58988)
0620 23:13:13.273953 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.273992 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260620 23:13:13.274070 23596 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c8d362c900e64b6d829a6ce9d0378a1b in current term 2: Already voted for candidate f235a868911c4de5b05381f909d97a75 in this term.
I20260620 23:13:13.273924 24108 leader_election.cc:288] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [CANDIDATE]: Term 2 election: Requested vote from peers 36ed237bf6274042870df583d008b757 (127.16.79.62:41873), f235a868911c4de5b05381f909d97a75 (127.16.79.60:46161)
I20260620 23:13:13.274540 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.274360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:59008)
0620 23:13:13.274457 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.274502 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 23:13:13.274636 23385 leader_election.cc:334] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 36ed237bf6274042870df583d008b757 (127.16.79.62:41873): Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
I20260620 23:13:13.274677 23385 leader_election.cc:302] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8d362c900e64b6d829a6ce9d0378a1b; no voters: 36ed237bf6274042870df583d008b757, f235a868911c4de5b05381f909d97a75
I20260620 23:13:13.274716 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.274540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:59010)
0620 23:13:13.274649 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.274686 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:13:13.274792 23572 leader_election.cc:334] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 36ed237bf6274042870df583d008b757 (127.16.79.62:41873): Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
I20260620 23:13:13.274844 23572 leader_election.cc:302] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f235a868911c4de5b05381f909d97a75; no voters: 36ed237bf6274042870df583d008b757, c8d362c900e64b6d829a6ce9d0378a1b
I20260620 23:13:13.274920 24106 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 23:13:13.274971 24108 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 23:13:13.764514 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.764163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.3:52087)
0620 23:13:13.764403 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.764466 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":123,"threads_started":1}
I20260620 23:13:13.767880 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.767534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.2:39483)
0620 23:13:13.767802 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.767846 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":179,"threads_started":1}
I20260620 23:13:13.767890 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:13.767460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.1:36243)
0620 23:13:13.767712 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:13.767831 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":89,"threads_started":1}
I20260620 23:13:14.765316 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:14.765057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.3:33843)
0620 23:13:14.765220 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:14.765256 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260620 23:13:14.768525 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:14.768347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.2:40001)
0620 23:13:14.768451 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:14.768481 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260620 23:13:14.768579 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:14.768390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.1:33065)
0620 23:13:14.768513 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:14.768547 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":54,"threads_started":1}
I20260620 23:13:15.071764 24122 raft_consensus.cc:482] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:15.071863 24122 raft_consensus.cc:504] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } }
I20260620 23:13:15.072080 24122 leader_election.cc:288] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36ed237bf6274042870df583d008b757 (127.16.79.62:41873), c8d362c900e64b6d829a6ce9d0378a1b (127.16.79.61:39717)
I20260620 23:13:15.072327 23409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f235a868911c4de5b05381f909d97a75" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" is_pre_election: true
I20260620 23:13:15.072434 23409 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f235a868911c4de5b05381f909d97a75 in term 2.
I20260620 23:13:15.072427 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.072232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:59234)
0620 23:13:15.072357 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.072390 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260620 23:13:15.072584 23570 leader_election.cc:302] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d362c900e64b6d829a6ce9d0378a1b, f235a868911c4de5b05381f909d97a75; no voters:
I20260620 23:13:15.072721 24122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 23:13:15.072768 24122 raft_consensus.cc:482] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:15.072806 24122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:13:15.072866 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.072786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:59246)
0620 23:13:15.072822 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.072845 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 23:13:15.072958 23572 leader_election.cc:334] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36ed237bf6274042870df583d008b757 (127.16.79.62:41873): Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
I20260620 23:13:15.073642 24122 raft_consensus.cc:504] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } }
I20260620 23:13:15.073761 24122 leader_election.cc:288] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [CANDIDATE]: Term 3 election: Requested vote from peers 36ed237bf6274042870df583d008b757 (127.16.79.62:41873), c8d362c900e64b6d829a6ce9d0378a1b (127.16.79.61:39717)
I20260620 23:13:15.073913 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.073834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:59260)
0620 23:13:15.073866 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.073894 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 23:13:15.073904 23409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f235a868911c4de5b05381f909d97a75" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8d362c900e64b6d829a6ce9d0378a1b"
I20260620 23:13:15.073992 23409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:13:15.074222 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.074113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:59264)
0620 23:13:15.074170 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.074201 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 23:13:15.074342 23572 leader_election.cc:334] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 36ed237bf6274042870df583d008b757 (127.16.79.62:41873): Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
I20260620 23:13:15.074707 23409 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f235a868911c4de5b05381f909d97a75 in term 3.
I20260620 23:13:15.074846 23570 leader_election.cc:302] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c8d362c900e64b6d829a6ce9d0378a1b, f235a868911c4de5b05381f909d97a75; no voters: 36ed237bf6274042870df583d008b757
I20260620 23:13:15.074923 24122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:13:15.075021 24122 raft_consensus.cc:686] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [term 3 LEADER]: Becoming Leader. State: Replica: f235a868911c4de5b05381f909d97a75, State: Running, Role: LEADER
I20260620 23:13:15.075105 24122 consensus_queue.cc:237] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } }
I20260620 23:13:15.075484 24125 sys_catalog.cc:455] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f235a868911c4de5b05381f909d97a75. Latest consensus state: current_term: 3 leader_uuid: "f235a868911c4de5b05381f909d97a75" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } } }
I20260620 23:13:15.075582 24125 sys_catalog.cc:458] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:15.075753 24127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:15.076148 24127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:15.076202 24127 catalog_manager.cc:1259] Loaded cluster ID: d50a1abbe97d4865bcba641d0523acfe
I20260620 23:13:15.076218 24127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:15.076349 24127 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:15.076404 24127 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: Loaded TSK: 0
I20260620 23:13:15.076488 24127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:13:15.104758 23585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:13:15.106240 23585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:13:15.107224 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.107117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:59274)
0620 23:13:15.107166 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.107191 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 23:13:15.107223 23409 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [term 3 FOLLOWER]: Refusing update from remote peer f235a868911c4de5b05381f909d97a75: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 23:13:15.107448 24122 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 23:13:15.107599 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.107492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.0.0.1:59282)
0620 23:13:15.107534 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.107577 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 23:13:15.107758 23572 consensus_peers.cc:487] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 -> Peer 36ed237bf6274042870df583d008b757 (127.16.79.62:41873): Couldn't send request to peer 36ed237bf6274042870df583d008b757. Status: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:13:15.108093 24129 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f235a868911c4de5b05381f909d97a75. Latest consensus state: current_term: 3 leader_uuid: "f235a868911c4de5b05381f909d97a75" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } } }
I20260620 23:13:15.108201 24129 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:13:15.108525 24128 mvcc.cc:205] Tried to move back new op lower bound from 7299060511157100544 to 7299060511028359168. Current Snapshot: MvccSnapshot[applied={T|T < 7299060511157100544}]
I20260620 23:13:15.108604 24125 sys_catalog.cc:455] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f235a868911c4de5b05381f909d97a75" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } } }
I20260620 23:13:15.108654 24130 mvcc.cc:205] Tried to move back new op lower bound from 7299060511157100544 to 7299060511028359168. Current Snapshot: MvccSnapshot[applied={T|T < 7299060511157100544}]
I20260620 23:13:15.108719 24125 sys_catalog.cc:458] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:15.108783 24129 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f235a868911c4de5b05381f909d97a75" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "36ed237bf6274042870df583d008b757" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41873 } } peers { permanent_uuid: "c8d362c900e64b6d829a6ce9d0378a1b" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39717 } attrs { promote: false } } peers { permanent_uuid: "f235a868911c4de5b05381f909d97a75" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46161 } attrs { promote: false } } }
I20260620 23:13:15.108881 24129 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d362c900e64b6d829a6ce9d0378a1b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:13:15.113855 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.110641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:46527 (local address 127.0.0.1:37338)
0620 23:13:15.110759 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.110768 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:15.110963 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:15.111176 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:15.111176 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:15.111178 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:15.111216 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:15.111217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.112076 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.112077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:15.112876 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:15.112878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.113604 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.113605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:15.113674 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:15.113678 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:15.113734 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 23:13:15.113780 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260620 23:13:15.113874 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.110693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.16.79.1:46527)
0620 23:13:15.110793 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:15.110795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:15.110804 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:15.111063 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:15.111064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:15.111074 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:15.111173 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:15.111292 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.112011 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.112958 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.113522 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.113596 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:15.113608 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:15.113755 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 23:13:15.113795 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":39,"threads_started":1}
I20260620 23:13:15.114197 23699 tablet_service.cc:1467] Processing CreateTablet for tablet 307abbf9c7794722aa954a384707b949 (DEFAULT_TABLE table=test-table [id=04d16c769ee2447686757b4d34444013]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:15.114300 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.110549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:44657 (local address 127.0.0.1:60176)
0620 23:13:15.110588 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.110599 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:15.110646 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:15.110867 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:15.110868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:15.110870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:15.110921 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:15.110922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.111865 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.111866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:15.113059 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:15.113061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.114111 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.114112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:15.114163 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:15.114166 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:13:15.114208 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 23:13:15.114240 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":27}
I20260620 23:13:15.114326 23699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 307abbf9c7794722aa954a384707b949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:15.114359 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.110632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.16.79.3:44657)
0620 23:13:15.110742 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:15.110744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:15.110752 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:13:15.110780 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:15.110781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:15.110792 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:15.110863 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:15.110996 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.111785 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.113133 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.113704 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.113790 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:15.113799 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:15.114232 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 23:13:15.114275 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260620 23:13:15.114974 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.110886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.16.79.2:40199)
0620 23:13:15.111901 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:15.111903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:15.111910 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:15.111933 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:15.111934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:15.111943 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:15.112000 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:15.112160 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.112911 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.114112 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.114660 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.114715 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:15.114724 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:15.114898 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 23:13:15.114926 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":52,"threads_started":1}
I20260620 23:13:15.115013 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.110822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:40199 (local address 127.0.0.1:36812)
0620 23:13:15.111059 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.111067 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:15.111100 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:15.112016 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:15.112017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:15.112019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:15.112059 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:15.112060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.112993 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.112994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:15.114023 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:15.114026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.114741 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.114742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:15.114819 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:15.114823 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:15.114888 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 23:13:15.114923 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":60,"threads_started":1}
I20260620 23:13:15.115600 24139 tablet_bootstrap.cc:492] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4: Bootstrap starting.
I20260620 23:13:15.115712 23972 tablet_service.cc:1467] Processing CreateTablet for tablet 307abbf9c7794722aa954a384707b949 (DEFAULT_TABLE table=test-table [id=04d16c769ee2447686757b4d34444013]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:15.115765 23835 tablet_service.cc:1467] Processing CreateTablet for tablet 307abbf9c7794722aa954a384707b949 (DEFAULT_TABLE table=test-table [id=04d16c769ee2447686757b4d34444013]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:15.115823 23972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 307abbf9c7794722aa954a384707b949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:15.115861 23835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 307abbf9c7794722aa954a384707b949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:15.115985 24139 tablet_bootstrap.cc:654] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:15.116777 24139 tablet_bootstrap.cc:492] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4: No bootstrap required, opened a new log
I20260620 23:13:15.116828 24139 ts_tablet_manager.cc:1397] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:15.116983 24139 raft_consensus.cc:348] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:15.117060 24139 raft_consensus.cc:374] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:15.117103 24139 raft_consensus.cc:729] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58731d03947949188070b9cc06fd12a4, State: Initialized, Role: FOLLOWER
I20260620 23:13:15.117173 24139 consensus_queue.cc:260] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:15.117239 24141 tablet_bootstrap.cc:492] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Bootstrap starting.
I20260620 23:13:15.117430 24139 ts_tablet_manager.cc:1428] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 23:13:15.117915 24141 tablet_bootstrap.cc:654] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:15.117727 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.117629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.1:59749)
0620 23:13:15.117668 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.117700 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 23:13:15.118068 23765 heartbeater.cc:503] Master 127.16.79.60:46161 was elected leader, sending a full tablet report...
I20260620 23:13:15.118257 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.118176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.1:53639)
0620 23:13:15.118203 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.118232 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
I20260620 23:13:15.118259 24143 tablet_bootstrap.cc:492] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d: Bootstrap starting.
I20260620 23:13:15.118669 24141 tablet_bootstrap.cc:492] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9: No bootstrap required, opened a new log
I20260620 23:13:15.118724 24141 ts_tablet_manager.cc:1397] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:13:15.118858 24141 raft_consensus.cc:348] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:15.118922 24141 raft_consensus.cc:374] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:15.118937 24141 raft_consensus.cc:729] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb4a8ac81a14379b3ba1bec6b8e25b9, State: Initialized, Role: FOLLOWER
I20260620 23:13:15.118968 24141 consensus_queue.cc:260] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:15.119076 24141 ts_tablet_manager.cc:1428] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:15.119325 24143 tablet_bootstrap.cc:654] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:15.119403 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.119310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.3:58665)
0620 23:13:15.119347 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.119378 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 23:13:15.119688 24038 heartbeater.cc:503] Master 127.16.79.60:46161 was elected leader, sending a full tablet report...
I20260620 23:13:15.119823 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.119635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.3:38283)
0620 23:13:15.119669 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.119697 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 23:13:15.120090 24143 tablet_bootstrap.cc:492] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d: No bootstrap required, opened a new log
I20260620 23:13:15.120154 24143 ts_tablet_manager.cc:1397] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:13:15.120239 24143 raft_consensus.cc:348] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:15.120287 24143 raft_consensus.cc:374] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:15.120301 24143 raft_consensus.cc:729] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb8acd7a81a4495889f5a4c3c83a47d, State: Initialized, Role: FOLLOWER
I20260620 23:13:15.120357 24143 consensus_queue.cc:260] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:15.120500 24143 ts_tablet_manager.cc:1428] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:15.120786 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.120674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.2:53807)
0620 23:13:15.120728 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.120758 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 23:13:15.121078 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.120977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.2:58715)
0620 23:13:15.121028 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.121055 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 23:13:15.121160 23902 heartbeater.cc:503] Master 127.16.79.60:46161 was elected leader, sending a full tablet report...
W20260620 23:13:15.319912 23768 tablet.cc:2369] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:13:15.325260 24142 raft_consensus.cc:482] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:15.325321 24142 raft_consensus.cc:504] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:15.325654 24142 leader_election.cc:288] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5eb4a8ac81a14379b3ba1bec6b8e25b9 (127.16.79.3:44657), edb8acd7a81a4495889f5a4c3c83a47d (127.16.79.2:40199)
I20260620 23:13:15.329694 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.325881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:37991 (local address 127.16.79.3:44657)
0620 23:13:15.325931 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:15.325933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:15.325946 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:13:15.325999 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:15.326000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:15.326013 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:15.326068 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:15.326272 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.327051 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.328546 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.329126 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.329204 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:15.329214 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:15.329591 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 23:13:15.329629 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:13:15.329694 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.325950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:38175 (local address 127.16.79.2:40199)
0620 23:13:15.325991 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:15.325993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:15.326138 (+ 145us) server_negotiation.cc:407] Connection header received
0620 23:13:15.326162 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:15.326163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:15.326173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:15.326214 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:15.326335 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.327051 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.328555 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.329126 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.329204 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:15.329214 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:15.329591 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 23:13:15.329630 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:13:15.329731 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.325857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:40199 (local address 127.16.79.1:38175)
0620 23:13:15.326032 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.326049 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:15.326115 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:15.326236 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:15.326237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:15.326238 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:15.326270 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:15.326271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.327143 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.327144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:15.328448 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:15.328451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.329308 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.329311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:15.329483 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:15.329493 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:13:15.329589 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:13:15.329625 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":32,"thread_start_us":113,"threads_started":1}
I20260620 23:13:15.329733 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.325812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:44657 (local address 127.16.79.1:37991)
0620 23:13:15.325867 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.325880 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:15.325938 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:15.326097 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:15.326098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:15.326101 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:15.326194 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:15.326198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.327145 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.327145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:15.328448 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:15.328451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.329309 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.329311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:15.329486 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:15.329491 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:13:15.329576 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:13:15.329625 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":88}
I20260620 23:13:15.330278 23856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307abbf9c7794722aa954a384707b949" candidate_uuid: "58731d03947949188070b9cc06fd12a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" is_pre_election: true
I20260620 23:13:15.330322 23992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307abbf9c7794722aa954a384707b949" candidate_uuid: "58731d03947949188070b9cc06fd12a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" is_pre_election: true
I20260620 23:13:15.330422 23856 raft_consensus.cc:2454] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58731d03947949188070b9cc06fd12a4 in term 0.
I20260620 23:13:15.330464 23992 raft_consensus.cc:2454] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58731d03947949188070b9cc06fd12a4 in term 0.
I20260620 23:13:15.330643 23654 leader_election.cc:302] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58731d03947949188070b9cc06fd12a4, edb8acd7a81a4495889f5a4c3c83a47d; no voters:
I20260620 23:13:15.330768 24142 raft_consensus.cc:2781] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:13:15.330822 24142 raft_consensus.cc:482] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:15.330848 24142 raft_consensus.cc:3037] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:15.331490 24142 raft_consensus.cc:504] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:15.331633 24142 leader_election.cc:288] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [CANDIDATE]: Term 1 election: Requested vote from peers 5eb4a8ac81a14379b3ba1bec6b8e25b9 (127.16.79.3:44657), edb8acd7a81a4495889f5a4c3c83a47d (127.16.79.2:40199)
I20260620 23:13:15.331804 23856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307abbf9c7794722aa954a384707b949" candidate_uuid: "58731d03947949188070b9cc06fd12a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb8acd7a81a4495889f5a4c3c83a47d"
I20260620 23:13:15.331849 23992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307abbf9c7794722aa954a384707b949" candidate_uuid: "58731d03947949188070b9cc06fd12a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9"
I20260620 23:13:15.331920 23992 raft_consensus.cc:3037] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:15.331919 23856 raft_consensus.cc:3037] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:15.332554 23992 raft_consensus.cc:2454] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58731d03947949188070b9cc06fd12a4 in term 1.
I20260620 23:13:15.332557 23856 raft_consensus.cc:2454] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58731d03947949188070b9cc06fd12a4 in term 1.
I20260620 23:13:15.332749 23655 leader_election.cc:302] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58731d03947949188070b9cc06fd12a4, 5eb4a8ac81a14379b3ba1bec6b8e25b9; no voters:
I20260620 23:13:15.332863 24142 raft_consensus.cc:2781] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:15.333001 24142 raft_consensus.cc:686] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [term 1 LEADER]: Becoming Leader. State: Replica: 58731d03947949188070b9cc06fd12a4, State: Running, Role: LEADER
I20260620 23:13:15.333081 24142 consensus_queue.cc:237] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } }
I20260620 23:13:15.333349 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.333265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.1:55031)
0620 23:13:15.333297 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.333324 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 23:13:15.333841 23585 catalog_manager.cc:5495] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58731d03947949188070b9cc06fd12a4 (127.16.79.1). New cstate: current_term: 1 leader_uuid: "58731d03947949188070b9cc06fd12a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58731d03947949188070b9cc06fd12a4" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 46527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 } health_report { overall_health: UNKNOWN } } }
W20260620 23:13:15.336613 24041 tablet.cc:2369] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:13:15.341810 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.339084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.16.79.1:46527)
0620 23:13:15.339135 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:15.339138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:15.339147 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:15.339191 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:15.339192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:15.339225 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:15.339314 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:13:15.339593 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.340093 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.340999 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:15.341201 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:15.341304 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:15.341483 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:13:15.341584 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:13:15.341672 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:15.341718 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 23:13:15.341739 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:13:15.342782 23992 raft_consensus.cc:1261] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 1 FOLLOWER]: Refusing update from remote peer 58731d03947949188070b9cc06fd12a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:15.342841 23856 raft_consensus.cc:1261] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d [term 1 FOLLOWER]: Refusing update from remote peer 58731d03947949188070b9cc06fd12a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:15.343050 24142 consensus_queue.cc:1035] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eb4a8ac81a14379b3ba1bec6b8e25b9" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:15.343191 24149 consensus_queue.cc:1035] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edb8acd7a81a4495889f5a4c3c83a47d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:15.344276 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.344179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.2:36587)
0620 23:13:15.344218 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.344244 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 23:13:15.344285 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.344095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.3:47621)
0620 23:13:15.344226 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.344255 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 23:13:15.345369 24153 mvcc.cc:205] Tried to move back new op lower bound from 7299060512121835520 to 7299060512084746240. Current Snapshot: MvccSnapshot[applied={T|T < 7299060512121835520}]
I20260620 23:13:15.345876 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.345779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.1:52381)
0620 23:13:15.345814 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.345845 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 23:13:15.345878 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.345779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.3:43239)
0620 23:13:15.345821 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.345845 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 23:13:15.346050 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.345944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.2:40341)
0620 23:13:15.346004 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.346027 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:13:15.346544 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.346365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:41873 (local address 127.16.79.1:46843)
0620 23:13:15.346484 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.346516 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 23:13:15.348141 23585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:15.348213 23585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:15.349514 23585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f235a868911c4de5b05381f909d97a75: Sending DeleteTablet for 3 replicas of tablet 307abbf9c7794722aa954a384707b949
I20260620 23:13:15.349788 23699 tablet_service.cc:1514] Processing DeleteTablet for tablet 307abbf9c7794722aa954a384707b949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:15 UTC) from {username='oryx', principal='oryx/127.16.79.60@KRBTEST.COM'} at 127.0.0.1:37338
I20260620 23:13:15.349822 23835 tablet_service.cc:1514] Processing DeleteTablet for tablet 307abbf9c7794722aa954a384707b949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:15 UTC) from {username='oryx', principal='oryx/127.16.79.60@KRBTEST.COM'} at 127.0.0.1:36812
I20260620 23:13:15.349874 23972 tablet_service.cc:1514] Processing DeleteTablet for tablet 307abbf9c7794722aa954a384707b949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:15 UTC) from {username='oryx', principal='oryx/127.16.79.60@KRBTEST.COM'} at 127.0.0.1:60176
I20260620 23:13:15.350040 24162 tablet_replica.cc:330] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4: stopping tablet replica
I20260620 23:13:15.350104 24162 raft_consensus.cc:2229] T 307abbf9c7794722aa954a384707b949 P 58731d03947949188070b9cc06fd12a4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:13:15.350207 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 23546
I20260620 23:13:15.352826 24163 tablet_replica.cc:330] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d: stopping tablet replica
I20260620 23:13:15.352902 24163 raft_consensus.cc:2229] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:15.352954 24163 raft_consensus.cc:2258] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:15.353060 24164 tablet_replica.cc:330] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9: stopping tablet replica
I20260620 23:13:15.353113 24164 raft_consensus.cc:2229] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:15.353180 24164 raft_consensus.cc:2258] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:15.353257 24163 ts_tablet_manager.cc:1905] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:15.353443 24164 ts_tablet_manager.cc:1905] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:15.354933 24164 ts_tablet_manager.cc:1918] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:15.354970 24164 log.cc:1196] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/307abbf9c7794722aa954a384707b949
I20260620 23:13:15.355139 24164 ts_tablet_manager.cc:1939] T 307abbf9c7794722aa954a384707b949 P 5eb4a8ac81a14379b3ba1bec6b8e25b9: Deleting consensus metadata
I20260620 23:13:15.355182 24163 ts_tablet_manager.cc:1918] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:15.355233 24163 log.cc:1196] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/307abbf9c7794722aa954a384707b949
I20260620 23:13:15.355479 23573 catalog_manager.cc:4841] TS 5eb4a8ac81a14379b3ba1bec6b8e25b9 (127.16.79.3:44657): tablet 307abbf9c7794722aa954a384707b949 (table test-table [id=04d16c769ee2447686757b4d34444013]) successfully deleted
I20260620 23:13:15.355723 24163 ts_tablet_manager.cc:1939] T 307abbf9c7794722aa954a384707b949 P edb8acd7a81a4495889f5a4c3c83a47d: Deleting consensus metadata
I20260620 23:13:15.356253 23572 catalog_manager.cc:4841] TS edb8acd7a81a4495889f5a4c3c83a47d (127.16.79.2:40199): tablet 307abbf9c7794722aa954a384707b949 (table test-table [id=04d16c769ee2447686757b4d34444013]) successfully deleted
I20260620 23:13:15.357563 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:15.357379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:46527 (local address 127.0.0.1:37356)
0620 23:13:15.357461 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:15.357505 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:46527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:13:15.357637 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 23773
W20260620 23:13:15.357720 23572 catalog_manager.cc:4584] TS 58731d03947949188070b9cc06fd12a4 (127.16.79.1:46527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 307abbf9c7794722aa954a384707b949: Network error: Client connection negotiation failed: client connection to 127.16.79.1:46527: connect: Connection refused (error 111)
I20260620 23:13:15.364766 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 23909
I20260620 23:13:15.369767 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 23266
I20260620 23:13:15.374679 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 23453
2026-06-20T23:13:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6596 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:13:15 dist-test-slave-pswb krb5kdc[24167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:13:15 dist-test-slave-pswb krb5kdc[24167](info): set up 2 sockets
Jun 20 23:13:15 dist-test-slave-pswb krb5kdc[24167](info): commencing operation
krb5kdc: starting...
W20260620 23:13:16.441601 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:13:16 dist-test-slave-pswb krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:13:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.62.keytab.
Entry for principal oryx/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.62.keytab.
I20260620 23:13:16.555397 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:34167
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.62.keytab
--principal=oryx/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:41255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:34167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:16.632120 24180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:16.632300 24180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:16.632328 24180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:16.633477 24180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:16.633517 24180 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:13:16.633530 24180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:16.633543 24180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:16.633556 24180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:16.633613 24180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:16.635025 24180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:41255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:34167
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.62.keytab
--principal=oryx/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:34167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:16.635243 24180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:16.635483 24180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:13:16.640308 24180 server_base.cc:1034] running on GCE node
W20260620 23:13:16.640264 24186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:16.640287 24188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:16.640322 24185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:16.640777 24180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:16.641016 24180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:16.642160 24180 hybrid_clock.cc:648] HybridClock initialized: now 1781997196642145 us; error 32 us; skew 500 ppm
Jun 20 23:13:16 dist-test-slave-pswb krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997196, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:16.644842 24180 init.cc:377] Logged in from keytab as oryx/127.16.79.62@KRBTEST.COM (short username oryx)
I20260620 23:13:16.645241 24180 webserver.cc:460] Webserver started at http://127.16.79.62:46467/ using document root <none> and password file <none>
I20260620 23:13:16.645503 24180 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:16.645569 24180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:16.645709 24180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:16.646696 24180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "dcbab62969e544d590897860477f52a1"
format_stamp: "Formatted at 2026-06-20 23:13:16 on dist-test-slave-pswb"
server_key: "71541bd36a113243d99912a838e7a20d"
server_key_iv: "eed4b5a58623ddb92cb3f6bc50f3daea"
server_key_version: "encryptionkey@0"
I20260620 23:13:16.647037 24180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "dcbab62969e544d590897860477f52a1"
format_stamp: "Formatted at 2026-06-20 23:13:16 on dist-test-slave-pswb"
server_key: "71541bd36a113243d99912a838e7a20d"
server_key_iv: "eed4b5a58623ddb92cb3f6bc50f3daea"
server_key_version: "encryptionkey@0"
I20260620 23:13:16.649174 24180 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:13:16.650163 24195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:16.650398 24180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:16.650499 24180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "dcbab62969e544d590897860477f52a1"
format_stamp: "Formatted at 2026-06-20 23:13:16 on dist-test-slave-pswb"
server_key: "71541bd36a113243d99912a838e7a20d"
server_key_iv: "eed4b5a58623ddb92cb3f6bc50f3daea"
server_key_version: "encryptionkey@0"
I20260620 23:13:16.650584 24180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:16.660370 24180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:16.662132 24180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:16.662274 24180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:16.666528 24180 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:34167
I20260620 23:13:16.666599 24257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:34167 every 8 connection(s)
I20260620 23:13:16.666888 24180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:16.667512 24258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:16.669425 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 24180
I20260620 23:13:16.669529 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:16.669699 24258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1: Bootstrap starting.
I20260620 23:13:16.669757 16700 external_mini_cluster.cc:1428] Setting key 5b7e31f9403b1869f3b3388212cd8827
I20260620 23:13:16.670383 24258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:16.670693 24258 log.cc:824] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:16.671489 24258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1: No bootstrap required, opened a new log
I20260620 23:13:16.672995 24258 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbab62969e544d590897860477f52a1" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34167 } }
I20260620 23:13:16.673125 24258 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:16.673146 24258 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbab62969e544d590897860477f52a1, State: Initialized, Role: FOLLOWER
I20260620 23:13:16.673290 24258 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbab62969e544d590897860477f52a1" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34167 } }
I20260620 23:13:16.673355 24258 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:16.673377 24258 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:16.673468 24258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:16.674028 24258 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbab62969e544d590897860477f52a1" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34167 } }
I20260620 23:13:16.674144 24258 leader_election.cc:302] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcbab62969e544d590897860477f52a1; no voters:
I20260620 23:13:16.674292 24258 leader_election.cc:288] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:16.674368 24263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:16.674504 24263 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [term 1 LEADER]: Becoming Leader. State: Replica: dcbab62969e544d590897860477f52a1, State: Running, Role: LEADER
I20260620 23:13:16.674503 24258 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:16.674789 24263 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbab62969e544d590897860477f52a1" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34167 } }
I20260620 23:13:16.675266 24264 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcbab62969e544d590897860477f52a1. Latest consensus state: current_term: 1 leader_uuid: "dcbab62969e544d590897860477f52a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbab62969e544d590897860477f52a1" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34167 } } }
I20260620 23:13:16.675408 24264 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:16.675606 24272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:16.675248 24265 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcbab62969e544d590897860477f52a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbab62969e544d590897860477f52a1" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34167 } } }
I20260620 23:13:16.676091 24265 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:16.676156 24272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:13:16 dist-test-slave-pswb krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
I20260620 23:13:16.677739 24272 catalog_manager.cc:1347] Generated new cluster ID: d69c630f2c1e4ec4afe5839e30ad5429
I20260620 23:13:16.677802 24272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:16.680970 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:16.670220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.16.79.62:34167)
0620 23:13:16.670378 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:16.670385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:16.670409 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:13:16.670886 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:16.670908 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:16.670969 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:16.671115 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:16.671859 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:16.673346 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:16.674124 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:16.674787 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:16.674895 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:16.677722 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:16.677731 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:16.677731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:16.677740 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:16.679407 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:16.679767 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:16.679770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:16.679771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:16.679859 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:16.680061 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:16.680062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:16.680063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:16.680291 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:16.680451 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:16.680534 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 23:13:16.680663 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260620 23:13:16.696477 24272 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:16.697052 24272 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:16.703637 24272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1: Generated new TSK 0
I20260620 23:13:16.703816 24272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab.
Entry for principal oryx/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab.
I20260620 23:13:16.729168 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34167
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab
--principal=oryx/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:41255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:16.808722 24284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:16.808912 24284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:16.808938 24284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:16.810055 24284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:16.810096 24284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:16.810117 24284 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:13:16.810134 24284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:16.811570 24284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:41255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab
--principal=oryx/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.79.62:34167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:16.811882 24284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:16.812103 24284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:16.816951 24290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:16.816951 24292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:16.816990 24289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:16.817313 24284 server_base.cc:1034] running on GCE node
I20260620 23:13:16.817657 24284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:16.817898 24284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:16.819079 24284 hybrid_clock.cc:648] HybridClock initialized: now 1781997196819074 us; error 40 us; skew 500 ppm
Jun 20 23:13:16 dist-test-slave-pswb krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997196, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:16.821609 24284 init.cc:377] Logged in from keytab as oryx/127.16.79.1@KRBTEST.COM (short username oryx)
I20260620 23:13:16.822000 24284 webserver.cc:460] Webserver started at http://127.16.79.1:33311/ using document root <none> and password file <none>
I20260620 23:13:16.822220 24284 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:16.822292 24284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:16.822436 24284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:16.823422 24284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "b90e1afca8dd4b27a7c253fe19c8e11e"
format_stamp: "Formatted at 2026-06-20 23:13:16 on dist-test-slave-pswb"
server_key: "e34ebe63769163159037492ee19fb3de"
server_key_iv: "a663c25b4d2bb6b2cba4e97660e5d24f"
server_key_version: "encryptionkey@0"
I20260620 23:13:16.823765 24284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "b90e1afca8dd4b27a7c253fe19c8e11e"
format_stamp: "Formatted at 2026-06-20 23:13:16 on dist-test-slave-pswb"
server_key: "e34ebe63769163159037492ee19fb3de"
server_key_iv: "a663c25b4d2bb6b2cba4e97660e5d24f"
server_key_version: "encryptionkey@0"
I20260620 23:13:16.825085 24284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 23:13:16.825870 24299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:16.826059 24284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:16.826152 24284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "b90e1afca8dd4b27a7c253fe19c8e11e"
format_stamp: "Formatted at 2026-06-20 23:13:16 on dist-test-slave-pswb"
server_key: "e34ebe63769163159037492ee19fb3de"
server_key_iv: "a663c25b4d2bb6b2cba4e97660e5d24f"
server_key_version: "encryptionkey@0"
I20260620 23:13:16.826231 24284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:16.847473 24284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:16.849251 24284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:16.849406 24284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:16.849848 24284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:16.849905 24284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:16.849937 24284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:16.849967 24284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:16.857088 24284 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:43675
Jun 20 23:13:16 dist-test-slave-pswb krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997196, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
I20260620 23:13:16.857169 24414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:43675 every 8 connection(s)
I20260620 23:13:16.857509 24284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:16.860060 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:16.850453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:49171 (local address 127.16.79.62:34167)
0620 23:13:16.850497 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:16.850499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:16.850695 (+ 196us) server_negotiation.cc:407] Connection header received
0620 23:13:16.851233 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:16.851234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:16.851248 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:16.851339 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:16.852347 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:16.853193 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:16.854820 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:16.855132 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:16.855205 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:16.857652 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:16.857654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:16.857655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:16.857662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:16.858863 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:16.859224 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:16.859225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:16.859225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:16.859274 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:16.859496 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:16.859497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:16.859497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:16.859598 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:16.859650 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:16.859931 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:13:16.859998 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:13:16.860306 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:16.850459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34167 (local address 127.16.79.1:49171)
0620 23:13:16.850587 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:16.850618 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:16.851087 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:16.851535 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:16.851540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:16.851942 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:16.852225 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:16.852230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:16.853353 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:16.853355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:16.854704 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:16.854710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:16.855221 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:16.855223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:16.855359 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:16.855893 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:16.855902 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:16.857546 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:16.858932 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:16.858936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:16.858938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:16.859156 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:16.859339 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:16.859339 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:16.859339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:16.859427 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:16.859677 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:16.859682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:16.859790 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 23:13:16.859903 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:13:16.860042 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20260620 23:13:16.860777 24415 heartbeater.cc:347] Connected to a master server at 127.16.79.62:34167
I20260620 23:13:16.860864 24415 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:16.861073 24415 heartbeater.cc:511] Master 127.16.79.62:34167 requested a full tablet report, sending...
I20260620 23:13:16.861852 24212 ts_manager.cc:194] Registered new tserver with Master: b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675)
I20260620 23:13:16.862624 24212 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:49171
I20260620 23:13:16.863853 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 24284
I20260620 23:13:16.863966 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:13:16.864174 16700 external_mini_cluster.cc:1428] Setting key c96494495cbb493fba1d6304cbb599f4
I20260620 23:13:16.869019 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:16.861814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:47333 (local address 127.16.79.62:34167)
0620 23:13:16.861857 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:16.861859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:16.861944 (+ 85us) server_negotiation.cc:407] Connection header received
0620 23:13:16.861985 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:16.861986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:16.861995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:16.862097 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:16.862360 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:16.862953 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:16.863859 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:16.864112 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:16.864171 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:16.865290 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:16.865293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:16.865293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:16.865303 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:16.867519 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:16.867863 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:16.867864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:16.867864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:16.867947 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:16.868163 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:16.868164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:16.868164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:16.868274 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:16.868453 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:16.868798 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 23:13:16.868907 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:13:16.869059 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:16.861752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34167 (local address 127.16.79.1:47333)
0620 23:13:16.861858 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:16.861868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:16.861924 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:16.862098 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:16.862099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:16.862242 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:16.862288 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:16.862289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:16.863022 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:16.863023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:16.863754 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:16.863757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:16.864187 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:16.864188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:16.864258 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:16.864660 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:16.864664 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:16.865055 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:16.867620 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:16.867620 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:16.867621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:16.867782 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:16.868019 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:16.868020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:16.868020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:16.868088 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:16.868488 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:16.868490 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:16.868588 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 23:13:16.868759 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 23:13:16.868931 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for oryx/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab.
Entry for principal oryx/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab.
I20260620 23:13:16.892220 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34167
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab
--principal=oryx/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:41255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:16.972975 24424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:16.973155 24424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:16.973179 24424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:16.974241 24424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:16.974275 24424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:16.974295 24424 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:13:16.974318 24424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:16.975840 24424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:41255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab
--principal=oryx/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.79.62:34167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:16.976043 24424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:16.976231 24424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:16.980839 24430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:16.980815 24432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:16.980826 24429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:16.981720 24424 server_base.cc:1034] running on GCE node
I20260620 23:13:16.982101 24424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:16.982384 24424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:16.983572 24424 hybrid_clock.cc:648] HybridClock initialized: now 1781997196983547 us; error 34 us; skew 500 ppm
Jun 20 23:13:16 dist-test-slave-pswb krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997196, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:16.986366 24424 init.cc:377] Logged in from keytab as oryx/127.16.79.2@KRBTEST.COM (short username oryx)
I20260620 23:13:16.986835 24424 webserver.cc:460] Webserver started at http://127.16.79.2:36589/ using document root <none> and password file <none>
I20260620 23:13:16.987071 24424 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:16.987114 24424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:16.987267 24424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:16.988255 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "7d70399e26494d619651210472ad8718"
format_stamp: "Formatted at 2026-06-20 23:13:16 on dist-test-slave-pswb"
server_key: "dd61a9fb5096cf43decd6e38cb70e8ca"
server_key_iv: "89abd7442aaf5afc0b16889e4ac78dcd"
server_key_version: "encryptionkey@0"
I20260620 23:13:16.988600 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "7d70399e26494d619651210472ad8718"
format_stamp: "Formatted at 2026-06-20 23:13:16 on dist-test-slave-pswb"
server_key: "dd61a9fb5096cf43decd6e38cb70e8ca"
server_key_iv: "89abd7442aaf5afc0b16889e4ac78dcd"
server_key_version: "encryptionkey@0"
I20260620 23:13:16.989883 24424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:16.990607 24439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:16.990773 24424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:16.990868 24424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "7d70399e26494d619651210472ad8718"
format_stamp: "Formatted at 2026-06-20 23:13:16 on dist-test-slave-pswb"
server_key: "dd61a9fb5096cf43decd6e38cb70e8ca"
server_key_iv: "89abd7442aaf5afc0b16889e4ac78dcd"
server_key_version: "encryptionkey@0"
I20260620 23:13:16.990948 24424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:17.006565 24424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:17.008853 24424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:17.009008 24424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:17.009572 24424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:17.009619 24424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:17.009650 24424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:17.009668 24424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:13:17 dist-test-slave-pswb krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997196, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
I20260620 23:13:17.016650 24424 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:37081
I20260620 23:13:17.016986 24424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:13:17.017040 24554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:37081 every 8 connection(s)
I20260620 23:13:17.018174 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 24424
I20260620 23:13:17.018354 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:17.018757 16700 external_mini_cluster.cc:1428] Setting key f74b83d17abce569f4e74412e15ac2e0
I20260620 23:13:17.019618 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.009995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:38993 (local address 127.16.79.62:34167)
0620 23:13:17.010069 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.010071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.010239 (+ 168us) server_negotiation.cc:407] Connection header received
0620 23:13:17.010678 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.010679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.010692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.010759 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:17.011354 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.012186 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.013239 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.013503 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.013567 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.016473 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:17.016474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:17.016475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:17.016481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:17.018133 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.018617 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.018618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.018619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.018674 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.018946 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.018947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.018947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.019097 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:17.019164 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.019482 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.019555 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:13:17.019898 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.009965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34167 (local address 127.16.79.2:38993)
0620 23:13:17.010105 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.010146 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.010589 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.010825 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.010828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.011066 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:17.011234 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.011237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.012270 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.012271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.013145 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.013147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.013599 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.013601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.013780 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.014426 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:17.014437 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:17.016375 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:17.018230 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:17.018236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:17.018238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:17.018521 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:17.018748 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:17.018749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:17.018749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:17.018891 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:17.019172 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:17.019178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:17.019322 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.019462 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.019631 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260620 23:13:17.021067 24555 heartbeater.cc:347] Connected to a master server at 127.16.79.62:34167
I20260620 23:13:17.021165 24555 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:17.021322 24555 heartbeater.cc:511] Master 127.16.79.62:34167 requested a full tablet report, sending...
I20260620 23:13:17.021641 24212 ts_manager.cc:194] Registered new tserver with Master: 7d70399e26494d619651210472ad8718 (127.16.79.2:37081)
I20260620 23:13:17.022158 24212 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:38993
I20260620 23:13:17.027230 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.021265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34167 (local address 127.16.79.2:54909)
0620 23:13:17.021405 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.021415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.021474 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.021596 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.021597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.021826 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:17.021874 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.021876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.022776 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.022777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.023379 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.023382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.023887 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.023888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.023939 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.024357 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:17.024362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:17.024612 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:17.026293 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:17.026294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:17.026294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:17.026447 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:17.026646 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:17.026646 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:17.026647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:17.026696 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:17.027004 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:17.027005 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:17.027037 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.027090 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.027162 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260620 23:13:17.027411 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.021310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:54909 (local address 127.16.79.62:34167)
0620 23:13:17.021350 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.021352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.021460 (+ 108us) server_negotiation.cc:407] Connection header received
0620 23:13:17.021529 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.021529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.021538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.021590 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:17.021951 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.022670 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.023467 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.023786 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.023863 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.024733 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:17.024735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:17.024735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:17.024742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:17.026196 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.026522 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.026523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.026523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.026576 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.026756 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.026757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.026757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.026935 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:17.027011 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.027120 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.027186 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for oryx/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab.
Entry for principal oryx/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab.
I20260620 23:13:17.043730 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34167
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab
--principal=oryx/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:41255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:17.123117 24564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:17.123297 24564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:17.123327 24564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:17.124445 24564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:17.124495 24564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:17.124516 24564 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:13:17.124533 24564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:17.126020 24564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:41255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab
--principal=oryx/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.79.62:34167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:17.126232 24564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:17.126451 24564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:17.131410 24569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:17.131469 24564 server_base.cc:1034] running on GCE node
W20260620 23:13:17.131520 24570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:17.131701 24572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:17.132257 24564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:17.132529 24564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:17.133710 24564 hybrid_clock.cc:648] HybridClock initialized: now 1781997197133693 us; error 47 us; skew 500 ppm
Jun 20 23:13:17 dist-test-slave-pswb krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997197, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:17.136193 24564 init.cc:377] Logged in from keytab as oryx/127.16.79.3@KRBTEST.COM (short username oryx)
I20260620 23:13:17.136626 24564 webserver.cc:460] Webserver started at http://127.16.79.3:37015/ using document root <none> and password file <none>
I20260620 23:13:17.136885 24564 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:17.136931 24564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:17.137017 24564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:17.137920 24564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "1d225484bf8545a1b856d5c7e2de01b4"
format_stamp: "Formatted at 2026-06-20 23:13:17 on dist-test-slave-pswb"
server_key: "102acd6e37ce7baeb2b3053194c4fc52"
server_key_iv: "44f11bc86060b689c6aff64a26144e06"
server_key_version: "encryptionkey@0"
I20260620 23:13:17.138199 24564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "1d225484bf8545a1b856d5c7e2de01b4"
format_stamp: "Formatted at 2026-06-20 23:13:17 on dist-test-slave-pswb"
server_key: "102acd6e37ce7baeb2b3053194c4fc52"
server_key_iv: "44f11bc86060b689c6aff64a26144e06"
server_key_version: "encryptionkey@0"
I20260620 23:13:17.139415 24564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:13:17.140079 24579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:17.140266 24564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:17.140342 24564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "1d225484bf8545a1b856d5c7e2de01b4"
format_stamp: "Formatted at 2026-06-20 23:13:17 on dist-test-slave-pswb"
server_key: "102acd6e37ce7baeb2b3053194c4fc52"
server_key_iv: "44f11bc86060b689c6aff64a26144e06"
server_key_version: "encryptionkey@0"
I20260620 23:13:17.140393 24564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:17.153081 24564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:17.155296 24564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:17.155398 24564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:17.155936 24564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:17.155992 24564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:17.156035 24564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:17.156061 24564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:13:17 dist-test-slave-pswb krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997197, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
I20260620 23:13:17.163691 24564 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:34751
I20260620 23:13:17.163739 24694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:34751 every 8 connection(s)
I20260620 23:13:17.164065 24564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:13:17.166129 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.156227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:53665 (local address 127.16.79.62:34167)
0620 23:13:17.156278 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.156281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.156508 (+ 227us) server_negotiation.cc:407] Connection header received
0620 23:13:17.157165 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.157166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.157180 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.157252 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:17.158056 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.158899 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.160405 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.160862 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.161151 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.163620 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:17.163622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:17.163622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:17.163629 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:17.165093 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.165434 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.165435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.165435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.165482 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.165653 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.165653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.165654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.165748 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:17.165796 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.166014 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.166071 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:13:17.166265 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.156221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34167 (local address 127.16.79.3:53665)
0620 23:13:17.156395 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.156431 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.157067 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.157314 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.157316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.157697 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:17.157956 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.157962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.159006 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.159008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.160253 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.160256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.161166 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.161167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.161328 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.161842 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:17.161849 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:17.163531 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:17.165163 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:17.165167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:17.165168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:17.165369 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:17.165522 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:17.165522 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:17.165522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:17.165598 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:17.165804 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:17.165808 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:17.165910 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.165988 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.166093 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260620 23:13:17.166723 24695 heartbeater.cc:347] Connected to a master server at 127.16.79.62:34167
I20260620 23:13:17.166826 24695 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:17.166996 24695 heartbeater.cc:511] Master 127.16.79.62:34167 requested a full tablet report, sending...
I20260620 23:13:17.167387 24212 ts_manager.cc:194] Registered new tserver with Master: 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751)
I20260620 23:13:17.167887 24212 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:53665
I20260620 23:13:17.167958 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 24564
I20260620 23:13:17.168064 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:17.168274 16700 external_mini_cluster.cc:1428] Setting key 3a00e7441de4518498992f1bbeeed678
I20260620 23:13:17.169090 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:13:17.173349 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.167746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:51603 (local address 127.16.79.62:34167)
0620 23:13:17.167808 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.167810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.167850 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:13:17.167920 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.167921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.167930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.167975 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:17.168397 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.168984 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.169635 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.169851 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.169936 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.170820 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:17.170821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:17.170822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:17.170828 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:17.172181 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.172562 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.172563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.172564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.172613 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.172860 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.172861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.172861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.172966 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:17.173040 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.173196 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.173265 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:13:17.173357 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.167676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34167 (local address 127.16.79.3:51603)
0620 23:13:17.167795 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.167806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.167866 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.168013 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.168014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.168215 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:17.168287 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.168289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.169074 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.169074 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.169550 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.169551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.169940 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.169941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.170016 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.170439 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:17.170443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:17.170713 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:17.172282 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:17.172283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:17.172284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:17.172450 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:17.172724 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:17.172725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:17.172726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:17.172782 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:17.173032 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:17.173033 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:17.173076 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.173170 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.173267 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
Jun 20 23:13:17 dist-test-slave-pswb krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997197, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:13:17 dist-test-slave-pswb krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:17 dist-test-slave-pswb krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:13:17.190263 24261 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:17.184954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.16.79.62:34167)
0620 23:13:17.185018 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.185020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.185027 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:17.185093 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.185094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.185101 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.185149 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:17.185643 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.186086 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.186746 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.186915 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.186968 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.190203 (+ 3235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54410: BlockingRecv error: recv got EOF from 127.0.0.1:54410 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 23:13:17 dist-test-slave-pswb krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997197, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
I20260620 23:13:17.199699 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.191918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.16.79.62:34167)
0620 23:13:17.192038 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.192040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.192050 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:13:17.192078 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.192078 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.192088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.192152 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:17.192619 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.193159 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.194208 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.194454 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.194570 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.196989 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:17.196991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:17.196991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:17.196999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:17.198568 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.198943 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.198944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.198944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.198995 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.199176 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.199177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.199177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.199284 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:17.199351 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.199535 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.199606 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:13:17.205389 24212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:13:17.207269 24212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:13:17.212957 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.209476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:37081 (local address 127.0.0.1:42130)
0620 23:13:17.209801 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.209811 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.209895 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.210114 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.210117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.210119 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:17.210181 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.210183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.211097 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.211098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.211889 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.211891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.212614 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.212615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.212737 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.212755 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.212822 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.212877 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260620 23:13:17.212968 24719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.209598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.16.79.2:37081)
0620 23:13:17.209844 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.209847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.209882 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:13:17.209983 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.209990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.210004 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.210114 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:17.210243 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.211020 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.211989 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.212536 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.212608 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.212678 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.212836 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.212878 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":74,"threads_started":1}
I20260620 23:13:17.213707 24489 tablet_service.cc:1467] Processing CreateTablet for tablet affddeefd9a1431cb18bfbbe49e5e37a (DEFAULT_TABLE table=test-table [id=e155e0314bdd4a81b4241e7f88d6766b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:17.213853 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.209476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:43675 (local address 127.0.0.1:38352)
0620 23:13:17.209768 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.209809 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.209884 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.210133 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.210145 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.210147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:17.210463 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.210465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.211315 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.211316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.212698 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.212699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.213613 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.213614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.213679 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.213683 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.213754 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.213794 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":77,"threads_started":1}
I20260620 23:13:17.213914 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.209598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.16.79.1:43675)
0620 23:13:17.209772 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.209777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.209968 (+ 191us) server_negotiation.cc:407] Connection header received
0620 23:13:17.210006 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.210014 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.210029 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.210122 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:17.210556 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.211226 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.212793 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.213531 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.213603 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.213669 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.213767 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.213805 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260620 23:13:17.214061 24489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet affddeefd9a1431cb18bfbbe49e5e37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:17.214546 24348 tablet_service.cc:1467] Processing CreateTablet for tablet affddeefd9a1431cb18bfbbe49e5e37a (DEFAULT_TABLE table=test-table [id=e155e0314bdd4a81b4241e7f88d6766b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:17.214845 24348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet affddeefd9a1431cb18bfbbe49e5e37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:17.216038 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.209786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.16.79.3:34751)
0620 23:13:17.209958 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.209962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.210315 (+ 353us) server_negotiation.cc:407] Connection header received
0620 23:13:17.210356 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.210358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.210374 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.210452 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:17.211348 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.212395 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.214628 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.215402 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.215480 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.215548 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.215903 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.215946 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260620 23:13:17.216336 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.209686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:34751 (local address 127.0.0.1:46924)
0620 23:13:17.210203 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.210213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.210259 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.210465 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.210466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.210468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:17.210503 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.210504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.212478 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.212479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.214528 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.214531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.215632 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.215635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.215761 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.215770 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.215869 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.215938 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":62,"threads_started":1}
I20260620 23:13:17.217378 24628 tablet_service.cc:1467] Processing CreateTablet for tablet affddeefd9a1431cb18bfbbe49e5e37a (DEFAULT_TABLE table=test-table [id=e155e0314bdd4a81b4241e7f88d6766b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:17.217690 24628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet affddeefd9a1431cb18bfbbe49e5e37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:17.218547 24723 tablet_bootstrap.cc:492] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718: Bootstrap starting.
I20260620 23:13:17.219137 24723 tablet_bootstrap.cc:654] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:17.219205 24724 tablet_bootstrap.cc:492] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e: Bootstrap starting.
I20260620 23:13:17.219574 24723 log.cc:824] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:17.219805 24724 tablet_bootstrap.cc:654] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:17.220153 24724 log.cc:824] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:17.220319 24723 tablet_bootstrap.cc:492] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718: No bootstrap required, opened a new log
I20260620 23:13:17.220381 24723 ts_tablet_manager.cc:1397] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:13:17.220988 24724 tablet_bootstrap.cc:492] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e: No bootstrap required, opened a new log
I20260620 23:13:17.221091 24724 ts_tablet_manager.cc:1397] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:13:17.221376 24727 tablet_bootstrap.cc:492] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4: Bootstrap starting.
I20260620 23:13:17.221865 24727 tablet_bootstrap.cc:654] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:17.222165 24727 log.cc:824] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:17.222378 24723 raft_consensus.cc:348] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.222582 24723 raft_consensus.cc:374] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:17.222623 24723 raft_consensus.cc:729] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d70399e26494d619651210472ad8718, State: Initialized, Role: FOLLOWER
I20260620 23:13:17.222731 24723 consensus_queue.cc:260] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.222838 24727 tablet_bootstrap.cc:492] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4: No bootstrap required, opened a new log
I20260620 23:13:17.222893 24727 ts_tablet_manager.cc:1397] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:13:17.222992 24723 ts_tablet_manager.cc:1428] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:13:17.223104 24555 heartbeater.cc:503] Master 127.16.79.62:34167 was elected leader, sending a full tablet report...
I20260620 23:13:17.223313 24724 raft_consensus.cc:348] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.223417 24724 raft_consensus.cc:374] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:17.223436 24724 raft_consensus.cc:729] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90e1afca8dd4b27a7c253fe19c8e11e, State: Initialized, Role: FOLLOWER
I20260620 23:13:17.223536 24724 consensus_queue.cc:260] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.223783 24724 ts_tablet_manager.cc:1428] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 23:13:17.223785 24415 heartbeater.cc:503] Master 127.16.79.62:34167 was elected leader, sending a full tablet report...
I20260620 23:13:17.224926 24727 raft_consensus.cc:348] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.225095 24727 raft_consensus.cc:374] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:17.225143 24727 raft_consensus.cc:729] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d225484bf8545a1b856d5c7e2de01b4, State: Initialized, Role: FOLLOWER
I20260620 23:13:17.225253 24727 consensus_queue.cc:260] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.225454 24727 ts_tablet_manager.cc:1428] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 23:13:17.225526 24695 heartbeater.cc:503] Master 127.16.79.62:34167 was elected leader, sending a full tablet report...
W20260620 23:13:17.267863 24556 tablet.cc:2369] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:13:17.301306 24730 raft_consensus.cc:482] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:17.301440 24730 raft_consensus.cc:504] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.301837 24730 leader_election.cc:288] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d70399e26494d619651210472ad8718 (127.16.79.2:37081), 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751)
I20260620 23:13:17.302759 24731 raft_consensus.cc:482] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:17.302870 24731 raft_consensus.cc:504] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.303306 24731 leader_election.cc:288] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d70399e26494d619651210472ad8718 (127.16.79.2:37081), b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675)
I20260620 23:13:17.305625 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.301983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:37081 (local address 127.16.79.1:47985)
0620 23:13:17.302067 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.302081 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.302149 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.302280 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.302281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.302282 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:17.302355 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.302357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.303308 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.303309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.304579 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.304582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.305338 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.305339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.305430 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.305434 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.305497 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.305538 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":49}
I20260620 23:13:17.305640 24719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.302036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:47985 (local address 127.16.79.2:37081)
0620 23:13:17.302089 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.302092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.302124 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:13:17.302217 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.302217 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.302227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.302273 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:17.302445 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.303212 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.304683 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.305244 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.305316 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.305336 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.305523 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.305573 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:13:17.305682 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.302055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:44719 (local address 127.16.79.3:34751)
0620 23:13:17.302099 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.302101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.302321 (+ 220us) server_negotiation.cc:407] Connection header received
0620 23:13:17.302352 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.302354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.302364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.302423 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:17.302568 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.303249 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.304775 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.305322 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.305360 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.305369 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.305590 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.305621 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:13:17.306015 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "affddeefd9a1431cb18bfbbe49e5e37a" candidate_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d70399e26494d619651210472ad8718" is_pre_election: true
I20260620 23:13:17.306093 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.301986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:34751 (local address 127.16.79.1:44719)
0620 23:13:17.302244 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.302252 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.302302 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.302441 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.302442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.302443 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:17.302497 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.302499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.303323 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.303324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.304668 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.304670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.305429 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.305430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.305507 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.305511 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.305941 (+ 430us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.305995 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":318,"thread_start_us":107,"threads_started":1}
I20260620 23:13:17.306159 24509 raft_consensus.cc:2454] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b90e1afca8dd4b27a7c253fe19c8e11e in term 0.
I20260620 23:13:17.306360 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "affddeefd9a1431cb18bfbbe49e5e37a" candidate_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d225484bf8545a1b856d5c7e2de01b4" is_pre_election: true
I20260620 23:13:17.306455 24649 raft_consensus.cc:2454] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b90e1afca8dd4b27a7c253fe19c8e11e in term 0.
I20260620 23:13:17.306697 24301 leader_election.cc:302] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d70399e26494d619651210472ad8718, b90e1afca8dd4b27a7c253fe19c8e11e; no voters:
I20260620 23:13:17.306871 24730 raft_consensus.cc:2781] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:13:17.306923 24730 raft_consensus.cc:482] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:17.306945 24730 raft_consensus.cc:3037] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:17.307523 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.303579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:42911 (local address 127.16.79.2:37081)
0620 23:13:17.303913 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.303915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.303922 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:17.303941 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.303942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.303949 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.304149 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:17.304181 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.304917 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.306346 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.307148 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.307225 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.307235 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.307406 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.307441 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":106,"threads_started":1}
I20260620 23:13:17.307523 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.303290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:37081 (local address 127.16.79.3:42911)
0620 23:13:17.303376 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.303386 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.303421 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.304042 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.304043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.304044 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:17.304088 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.304089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.305000 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.305001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.306084 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.306086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.307238 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.307239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.307308 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.307312 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.307389 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.307438 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":25}
I20260620 23:13:17.307753 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "affddeefd9a1431cb18bfbbe49e5e37a" candidate_uuid: "1d225484bf8545a1b856d5c7e2de01b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d70399e26494d619651210472ad8718" is_pre_election: true
I20260620 23:13:17.307819 24509 raft_consensus.cc:2454] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d225484bf8545a1b856d5c7e2de01b4 in term 0.
I20260620 23:13:17.307883 24730 raft_consensus.cc:504] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.308007 24581 leader_election.cc:302] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d225484bf8545a1b856d5c7e2de01b4, 7d70399e26494d619651210472ad8718; no voters:
I20260620 23:13:17.308122 24730 leader_election.cc:288] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [CANDIDATE]: Term 1 election: Requested vote from peers 7d70399e26494d619651210472ad8718 (127.16.79.2:37081), 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751)
I20260620 23:13:17.308133 24731 raft_consensus.cc:2781] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:13:17.308176 24731 raft_consensus.cc:482] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:17.308198 24731 raft_consensus.cc:3037] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:17.308276 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "affddeefd9a1431cb18bfbbe49e5e37a" candidate_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d225484bf8545a1b856d5c7e2de01b4"
I20260620 23:13:17.308707 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.303290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:43675 (local address 127.16.79.3:51975)
0620 23:13:17.303727 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.303736 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.303811 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.303984 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.303985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.303987 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:17.304051 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.304053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.304975 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.304976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.307677 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.307680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.308438 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.308438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.308506 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.308510 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.308576 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.308614 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":268,"thread_start_us":103,"threads_started":1}
I20260620 23:13:17.308727 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.303490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:51975 (local address 127.16.79.1:43675)
0620 23:13:17.303550 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.303552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.303807 (+ 255us) server_negotiation.cc:407] Connection header received
0620 23:13:17.303872 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.303873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.303895 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.303942 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:17.304132 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.304880 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.307775 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.308318 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.308361 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.308369 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.308587 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.308626 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:13:17.308740 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "affddeefd9a1431cb18bfbbe49e5e37a" candidate_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d70399e26494d619651210472ad8718"
I20260620 23:13:17.308804 24509 raft_consensus.cc:3037] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:17.309641 24509 raft_consensus.cc:2454] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b90e1afca8dd4b27a7c253fe19c8e11e in term 1.
I20260620 23:13:17.309798 24301 leader_election.cc:302] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d70399e26494d619651210472ad8718, b90e1afca8dd4b27a7c253fe19c8e11e; no voters:
I20260620 23:13:17.309808 24369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "affddeefd9a1431cb18bfbbe49e5e37a" candidate_uuid: "1d225484bf8545a1b856d5c7e2de01b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" is_pre_election: true
I20260620 23:13:17.309904 24730 raft_consensus.cc:2781] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:17.310019 24730 raft_consensus.cc:686] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 LEADER]: Becoming Leader. State: Replica: b90e1afca8dd4b27a7c253fe19c8e11e, State: Running, Role: LEADER
I20260620 23:13:17.310105 24730 consensus_queue.cc:237] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.310099 24731 raft_consensus.cc:504] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.310237 24649 raft_consensus.cc:2379] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b90e1afca8dd4b27a7c253fe19c8e11e in current term 1: Already voted for candidate 1d225484bf8545a1b856d5c7e2de01b4 in this term.
I20260620 23:13:17.310252 24731 leader_election.cc:288] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [CANDIDATE]: Term 1 election: Requested vote from peers 7d70399e26494d619651210472ad8718 (127.16.79.2:37081), b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675)
I20260620 23:13:17.310365 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "affddeefd9a1431cb18bfbbe49e5e37a" candidate_uuid: "1d225484bf8545a1b856d5c7e2de01b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d70399e26494d619651210472ad8718"
I20260620 23:13:17.310444 24509 raft_consensus.cc:2379] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1d225484bf8545a1b856d5c7e2de01b4 in current term 1: Already voted for candidate b90e1afca8dd4b27a7c253fe19c8e11e in this term.
I20260620 23:13:17.310576 24368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "affddeefd9a1431cb18bfbbe49e5e37a" candidate_uuid: "1d225484bf8545a1b856d5c7e2de01b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e"
I20260620 23:13:17.311229 24581 leader_election.cc:302] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d225484bf8545a1b856d5c7e2de01b4; no voters: 7d70399e26494d619651210472ad8718, b90e1afca8dd4b27a7c253fe19c8e11e
I20260620 23:13:17.311183 24211 catalog_manager.cc:5495] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e reported cstate change: term changed from 0 to 1, leader changed from <none> to b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1). New cstate: current_term: 1 leader_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } health_report { overall_health: UNKNOWN } } }
I20260620 23:13:17.311307 24731 raft_consensus.cc:2726] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 23:13:17.329228 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.326895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.16.79.1:43675)
0620 23:13:17.326939 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.326941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.326950 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:17.327005 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.327006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.327026 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.327095 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:13:17.327244 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.327716 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.328536 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.328772 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.328825 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.328950 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:13:17.329032 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:13:17.329087 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.329172 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.329186 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:13:17.330845 24649 raft_consensus.cc:1261] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Refusing update from remote peer b90e1afca8dd4b27a7c253fe19c8e11e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:17.330869 24509 raft_consensus.cc:1261] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Refusing update from remote peer b90e1afca8dd4b27a7c253fe19c8e11e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:17.331053 24730 consensus_queue.cc:1035] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:17.331169 24735 consensus_queue.cc:1035] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:17.333540 24738 mvcc.cc:205] Tried to move back new op lower bound from 7299060520265007104 to 7299060520182763520. Current Snapshot: MvccSnapshot[applied={T|T < 7299060520265007104}]
I20260620 23:13:17.333746 24739 mvcc.cc:205] Tried to move back new op lower bound from 7299060520265007104 to 7299060520182763520. Current Snapshot: MvccSnapshot[applied={T|T < 7299060520265007104}]
I20260620 23:13:17.335951 24211 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:17.336041 24211 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:17.336740 24211 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1: Sending DeleteTablet for 3 replicas of tablet affddeefd9a1431cb18bfbbe49e5e37a
I20260620 23:13:17.337087 24628 tablet_service.cc:1514] Processing DeleteTablet for tablet affddeefd9a1431cb18bfbbe49e5e37a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:17 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:46924
I20260620 23:13:17.337127 24348 tablet_service.cc:1514] Processing DeleteTablet for tablet affddeefd9a1431cb18bfbbe49e5e37a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:17 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:38352
I20260620 23:13:17.337139 24489 tablet_service.cc:1514] Processing DeleteTablet for tablet affddeefd9a1431cb18bfbbe49e5e37a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:17 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:42130
I20260620 23:13:17.337435 24750 tablet_replica.cc:330] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718: stopping tablet replica
I20260620 23:13:17.337532 24750 raft_consensus.cc:2229] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:17.337535 24211 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 23:13:17.337635 24750 raft_consensus.cc:2258] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 23:13:17.337678 24211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:13:17.338052 24750 ts_tablet_manager.cc:1905] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:17.338829 24628 tablet_service.cc:1467] Processing CreateTablet for tablet d4313a9d397745c78911fada6077323a (DEFAULT_TABLE table=test-table [id=e2fbbd8b5f9242458089da1e273d7fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:17.338922 24489 tablet_service.cc:1467] Processing CreateTablet for tablet d4313a9d397745c78911fada6077323a (DEFAULT_TABLE table=test-table [id=e2fbbd8b5f9242458089da1e273d7fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:17.338946 24628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4313a9d397745c78911fada6077323a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:17.339030 24489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4313a9d397745c78911fada6077323a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:17.339593 24750 ts_tablet_manager.cc:1918] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:17.339653 24750 log.cc:1196] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/affddeefd9a1431cb18bfbbe49e5e37a
I20260620 23:13:17.339900 24750 ts_tablet_manager.cc:1939] T affddeefd9a1431cb18bfbbe49e5e37a P 7d70399e26494d619651210472ad8718: Deleting consensus metadata
I20260620 23:13:17.340246 24723 tablet_bootstrap.cc:492] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718: Bootstrap starting.
I20260620 23:13:17.340261 24727 tablet_bootstrap.cc:492] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4: Bootstrap starting.
I20260620 23:13:17.340462 24197 catalog_manager.cc:4841] TS 7d70399e26494d619651210472ad8718 (127.16.79.2:37081): tablet affddeefd9a1431cb18bfbbe49e5e37a (table test-table [id=e155e0314bdd4a81b4241e7f88d6766b]) successfully deleted
I20260620 23:13:17.340605 24348 tablet_service.cc:1467] Processing CreateTablet for tablet d4313a9d397745c78911fada6077323a (DEFAULT_TABLE table=test-table [id=e2fbbd8b5f9242458089da1e273d7fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:17.340646 24727 tablet_bootstrap.cc:654] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:17.340646 24723 tablet_bootstrap.cc:654] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:17.340682 24348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4313a9d397745c78911fada6077323a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:17.340247 24748 tablet_replica.cc:330] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4: stopping tablet replica
I20260620 23:13:17.341050 24748 raft_consensus.cc:2229] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:17.341162 24748 raft_consensus.cc:2258] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:17.341449 24727 tablet_bootstrap.cc:492] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4: No bootstrap required, opened a new log
I20260620 23:13:17.341501 24727 ts_tablet_manager.cc:1397] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:17.341589 24723 tablet_bootstrap.cc:492] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718: No bootstrap required, opened a new log
I20260620 23:13:17.341634 24723 ts_tablet_manager.cc:1397] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:17.341650 24727 raft_consensus.cc:348] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.341723 24727 raft_consensus.cc:374] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:17.341751 24727 raft_consensus.cc:729] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d225484bf8545a1b856d5c7e2de01b4, State: Initialized, Role: FOLLOWER
I20260620 23:13:17.341789 24727 consensus_queue.cc:260] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.341794 24723 raft_consensus.cc:348] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.341849 24723 raft_consensus.cc:374] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:17.341876 24723 raft_consensus.cc:729] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d70399e26494d619651210472ad8718, State: Initialized, Role: FOLLOWER
I20260620 23:13:17.341883 24727 ts_tablet_manager.cc:1428] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:17.341917 24723 consensus_queue.cc:260] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.342011 24723 ts_tablet_manager.cc:1428] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:17.342080 24724 tablet_bootstrap.cc:492] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e: Bootstrap starting.
I20260620 23:13:17.342521 24724 tablet_bootstrap.cc:654] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:17.342639 24748 ts_tablet_manager.cc:1905] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:17.343107 24724 tablet_bootstrap.cc:492] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e: No bootstrap required, opened a new log
I20260620 23:13:17.343171 24724 ts_tablet_manager.cc:1397] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:17.343338 24724 raft_consensus.cc:348] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.343410 24724 raft_consensus.cc:374] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:17.343437 24724 raft_consensus.cc:729] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90e1afca8dd4b27a7c253fe19c8e11e, State: Initialized, Role: FOLLOWER
I20260620 23:13:17.343477 24724 consensus_queue.cc:260] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.343566 24724 ts_tablet_manager.cc:1428] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:17.344122 24748 ts_tablet_manager.cc:1918] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:17.344197 24748 log.cc:1196] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/affddeefd9a1431cb18bfbbe49e5e37a
I20260620 23:13:17.344440 24748 ts_tablet_manager.cc:1939] T affddeefd9a1431cb18bfbbe49e5e37a P 1d225484bf8545a1b856d5c7e2de01b4: Deleting consensus metadata
I20260620 23:13:17.344792 24199 catalog_manager.cc:4841] TS 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751): tablet affddeefd9a1431cb18bfbbe49e5e37a (table test-table [id=e155e0314bdd4a81b4241e7f88d6766b]) successfully deleted
I20260620 23:13:17.344856 24749 tablet_replica.cc:330] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e: stopping tablet replica
I20260620 23:13:17.344949 24749 raft_consensus.cc:2229] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:13:17.345046 24749 raft_consensus.cc:2258] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:17.345383 24749 ts_tablet_manager.cc:1905] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:17.346503 24749 ts_tablet_manager.cc:1918] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:17.346586 24749 log.cc:1196] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/wal/wals/affddeefd9a1431cb18bfbbe49e5e37a
I20260620 23:13:17.346812 24749 ts_tablet_manager.cc:1939] T affddeefd9a1431cb18bfbbe49e5e37a P b90e1afca8dd4b27a7c253fe19c8e11e: Deleting consensus metadata
I20260620 23:13:17.347198 24196 catalog_manager.cc:4841] TS b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675): tablet affddeefd9a1431cb18bfbbe49e5e37a (table test-table [id=e155e0314bdd4a81b4241e7f88d6766b]) successfully deleted
W20260620 23:13:17.358180 24416 tablet.cc:2369] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:13:17.414938 24696 tablet.cc:2369] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:13:17.569058 24730 raft_consensus.cc:482] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:17.569154 24730 raft_consensus.cc:504] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.569626 24730 leader_election.cc:288] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d70399e26494d619651210472ad8718 (127.16.79.2:37081), 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751)
I20260620 23:13:17.569751 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4313a9d397745c78911fada6077323a" candidate_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d70399e26494d619651210472ad8718" is_pre_election: true
I20260620 23:13:17.569883 24509 raft_consensus.cc:2454] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b90e1afca8dd4b27a7c253fe19c8e11e in term 0.
I20260620 23:13:17.570027 24301 leader_election.cc:302] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d70399e26494d619651210472ad8718, b90e1afca8dd4b27a7c253fe19c8e11e; no voters:
I20260620 23:13:17.570009 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4313a9d397745c78911fada6077323a" candidate_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d225484bf8545a1b856d5c7e2de01b4" is_pre_election: true
I20260620 23:13:17.570106 24649 raft_consensus.cc:2454] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b90e1afca8dd4b27a7c253fe19c8e11e in term 0.
I20260620 23:13:17.570156 24730 raft_consensus.cc:2781] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:13:17.570211 24730 raft_consensus.cc:482] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:17.570235 24730 raft_consensus.cc:3037] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:17.570921 24730 raft_consensus.cc:504] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.571075 24730 leader_election.cc:288] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [CANDIDATE]: Term 1 election: Requested vote from peers 7d70399e26494d619651210472ad8718 (127.16.79.2:37081), 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751)
I20260620 23:13:17.571300 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4313a9d397745c78911fada6077323a" candidate_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d70399e26494d619651210472ad8718"
I20260620 23:13:17.571269 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4313a9d397745c78911fada6077323a" candidate_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d225484bf8545a1b856d5c7e2de01b4"
I20260620 23:13:17.571384 24509 raft_consensus.cc:3037] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:17.571413 24649 raft_consensus.cc:3037] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:17.572003 24509 raft_consensus.cc:2454] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b90e1afca8dd4b27a7c253fe19c8e11e in term 1.
I20260620 23:13:17.572175 24301 leader_election.cc:302] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d70399e26494d619651210472ad8718, b90e1afca8dd4b27a7c253fe19c8e11e; no voters:
I20260620 23:13:17.572391 24730 raft_consensus.cc:2781] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:17.572454 24730 raft_consensus.cc:686] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 LEADER]: Becoming Leader. State: Replica: b90e1afca8dd4b27a7c253fe19c8e11e, State: Running, Role: LEADER
I20260620 23:13:17.572531 24730 consensus_queue.cc:237] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.572568 24649 raft_consensus.cc:2454] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b90e1afca8dd4b27a7c253fe19c8e11e in term 1.
I20260620 23:13:17.573076 24212 catalog_manager.cc:5495] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e reported cstate change: term changed from 0 to 1, leader changed from <none> to b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1). New cstate: current_term: 1 leader_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:13:17 dist-test-slave-pswb krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997196, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.62@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
I20260620 23:13:17.644022 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.637312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34167 (local address 127.0.0.1:54440)
0620 23:13:17.637488 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.637499 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.637559 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.637715 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.637716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.637987 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:17.638030 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.638032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.638630 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.638631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.639143 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.639144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.639440 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.639441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.639477 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.639926 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:17.639930 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:17.641414 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:17.642913 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:17.642919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:17.642920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:17.643113 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:17.643286 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:17.643286 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:17.643286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:17.643338 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:17.643630 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:17.643637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:17.643689 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.643755 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.643863 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
I20260620 23:13:17.644063 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.637370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.16.79.62:34167)
0620 23:13:17.637435 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.637438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.637574 (+ 136us) server_negotiation.cc:407] Connection header received
0620 23:13:17.637598 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.637599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.637610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.637713 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:17.638097 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.638565 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.639208 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.639380 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.639428 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.641484 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:17.641485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:17.641485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:17.641491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:17.642815 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.643178 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.643179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.643179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.643228 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.643412 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.643412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.643413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.643512 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:17.643565 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.643783 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.643909 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":40}
I20260620 23:13:17.645829 24212 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:54440:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:13:17.646044 24212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:13:17.647432 24628 tablet_service.cc:1467] Processing CreateTablet for tablet 7f02a0f5c11b4820a34f87c9975b3217 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ffa41a338c0249b490cfd06a6571e877]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:13:17.647426 24489 tablet_service.cc:1467] Processing CreateTablet for tablet 7f02a0f5c11b4820a34f87c9975b3217 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ffa41a338c0249b490cfd06a6571e877]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:13:17.647423 24348 tablet_service.cc:1467] Processing CreateTablet for tablet 7f02a0f5c11b4820a34f87c9975b3217 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ffa41a338c0249b490cfd06a6571e877]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:13:17.647586 24348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f02a0f5c11b4820a34f87c9975b3217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:17.647584 24489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f02a0f5c11b4820a34f87c9975b3217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:17.647589 24628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f02a0f5c11b4820a34f87c9975b3217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:17.648680 24727 tablet_bootstrap.cc:492] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4: Bootstrap starting.
I20260620 23:13:17.649032 24724 tablet_bootstrap.cc:492] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e: Bootstrap starting.
I20260620 23:13:17.649132 24727 tablet_bootstrap.cc:654] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:17.649338 24724 tablet_bootstrap.cc:654] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:17.649560 24723 tablet_bootstrap.cc:492] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: Bootstrap starting.
I20260620 23:13:17.649793 24727 tablet_bootstrap.cc:492] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4: No bootstrap required, opened a new log
I20260620 23:13:17.649812 24724 tablet_bootstrap.cc:492] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e: No bootstrap required, opened a new log
I20260620 23:13:17.649849 24724 ts_tablet_manager.cc:1397] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:17.649849 24727 ts_tablet_manager.cc:1397] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:17.649900 24723 tablet_bootstrap.cc:654] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:17.649960 24724 raft_consensus.cc:348] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.650022 24724 raft_consensus.cc:374] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:17.649993 24727 raft_consensus.cc:348] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.650043 24724 raft_consensus.cc:729] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90e1afca8dd4b27a7c253fe19c8e11e, State: Initialized, Role: FOLLOWER
I20260620 23:13:17.650061 24727 raft_consensus.cc:374] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:17.650084 24727 raft_consensus.cc:729] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d225484bf8545a1b856d5c7e2de01b4, State: Initialized, Role: FOLLOWER
I20260620 23:13:17.650092 24724 consensus_queue.cc:260] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.650125 24727 consensus_queue.cc:260] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.650189 24724 ts_tablet_manager.cc:1428] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:17.650208 24727 ts_tablet_manager.cc:1428] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:17.650207 24730 tablet_replica.cc:439] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } } }
I20260620 23:13:17.650296 24730 tablet_replica.cc:442] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:13:17.650305 24731 tablet_replica.cc:439] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } } }
I20260620 23:13:17.650380 24731 tablet_replica.cc:442] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:13:17.650689 24723 tablet_bootstrap.cc:492] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: No bootstrap required, opened a new log
I20260620 23:13:17.650743 24723 ts_tablet_manager.cc:1397] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:17.650878 24723 raft_consensus.cc:348] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.650965 24723 raft_consensus.cc:374] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:17.650993 24723 raft_consensus.cc:729] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d70399e26494d619651210472ad8718, State: Initialized, Role: FOLLOWER
I20260620 23:13:17.651034 24723 consensus_queue.cc:260] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.651121 24723 ts_tablet_manager.cc:1428] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:17.651161 24729 tablet_replica.cc:439] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } } }
I20260620 23:13:17.651244 24729 tablet_replica.cc:442] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:13:17.857942 24729 raft_consensus.cc:482] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:17.858038 24729 raft_consensus.cc:504] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.858397 24729 leader_election.cc:288] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675), 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751)
I20260620 23:13:17.861711 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.858677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:42071 (local address 127.16.79.1:43675)
0620 23:13:17.858827 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.858829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.858837 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:13:17.858866 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.858867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.858878 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.858949 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:17.859105 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.859866 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.860986 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.861369 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.861451 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.861463 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.861607 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.861648 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":39,"threads_started":1}
I20260620 23:13:17.861726 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.858739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:37975 (local address 127.16.79.3:34751)
0620 23:13:17.859021 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.859023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.859028 (+ 5us) server_negotiation.cc:407] Connection header received
0620 23:13:17.859048 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.859048 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.859056 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.859144 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:17.859235 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.859950 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.860972 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.861357 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.861451 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.861464 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.861610 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.861646 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":40,"threads_started":1}
I20260620 23:13:17.861721 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.858540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:43675 (local address 127.16.79.2:42071)
0620 23:13:17.858709 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.858721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.858774 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.858965 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.858966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.858968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:17.859024 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.859026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.859950 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.859951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.860892 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.860894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.861453 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.861454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.861523 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.861526 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.861589 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.861631 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":31,"thread_start_us":70,"threads_started":1}
I20260620 23:13:17.861726 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.858556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:34751 (local address 127.16.79.2:37975)
0620 23:13:17.858765 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.858776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.858833 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.859143 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.859144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.859145 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:17.859170 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.859171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.860016 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.860017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.860878 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.860880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.861456 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.861457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.861526 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.861529 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.861604 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.861646 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":53,"threads_started":1}
I20260620 23:13:17.862187 24369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f02a0f5c11b4820a34f87c9975b3217" candidate_uuid: "7d70399e26494d619651210472ad8718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" is_pre_election: true
I20260620 23:13:17.862182 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f02a0f5c11b4820a34f87c9975b3217" candidate_uuid: "7d70399e26494d619651210472ad8718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d225484bf8545a1b856d5c7e2de01b4" is_pre_election: true
I20260620 23:13:17.862257 24649 raft_consensus.cc:2454] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d70399e26494d619651210472ad8718 in term 0.
I20260620 23:13:17.862282 24369 raft_consensus.cc:2454] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d70399e26494d619651210472ad8718 in term 0.
I20260620 23:13:17.862411 24443 leader_election.cc:302] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d225484bf8545a1b856d5c7e2de01b4, 7d70399e26494d619651210472ad8718; no voters:
I20260620 23:13:17.862537 24729 raft_consensus.cc:2781] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:13:17.862581 24729 raft_consensus.cc:482] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:17.862612 24729 raft_consensus.cc:3037] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:17.863240 24729 raft_consensus.cc:504] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.863365 24729 leader_election.cc:288] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [CANDIDATE]: Term 1 election: Requested vote from peers b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675), 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751)
I20260620 23:13:17.863499 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f02a0f5c11b4820a34f87c9975b3217" candidate_uuid: "7d70399e26494d619651210472ad8718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d225484bf8545a1b856d5c7e2de01b4"
I20260620 23:13:17.863564 24649 raft_consensus.cc:3037] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:17.863582 24369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f02a0f5c11b4820a34f87c9975b3217" candidate_uuid: "7d70399e26494d619651210472ad8718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e"
I20260620 23:13:17.863668 24369 raft_consensus.cc:3037] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:17.864135 24649 raft_consensus.cc:2454] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d70399e26494d619651210472ad8718 in term 1.
I20260620 23:13:17.864296 24443 leader_election.cc:302] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d225484bf8545a1b856d5c7e2de01b4, 7d70399e26494d619651210472ad8718; no voters:
I20260620 23:13:17.864389 24729 raft_consensus.cc:2781] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:17.864401 24369 raft_consensus.cc:2454] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d70399e26494d619651210472ad8718 in term 1.
I20260620 23:13:17.864585 24729 raft_consensus.cc:686] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [term 1 LEADER]: Becoming Leader. State: Replica: 7d70399e26494d619651210472ad8718, State: Running, Role: LEADER
I20260620 23:13:17.864657 24729 consensus_queue.cc:237] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.864981 24768 tablet_replica.cc:439] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: TxnStatusTablet state changed. Reason: New leader 7d70399e26494d619651210472ad8718. Latest consensus state: current_term: 1 leader_uuid: "7d70399e26494d619651210472ad8718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } } }
I20260620 23:13:17.865113 24768 tablet_replica.cc:442] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:13:17.865290 24770 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:13:17.865373 24770 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:13:17.865423 24211 catalog_manager.cc:5495] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d70399e26494d619651210472ad8718 (127.16.79.2). New cstate: current_term: 1 leader_uuid: "7d70399e26494d619651210472ad8718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } health_report { overall_health: UNKNOWN } } }
I20260620 23:13:17.878041 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.875392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42134 (local address 127.16.79.2:37081)
0620 23:13:17.875509 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.875511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.875564 (+ 53us) server_negotiation.cc:407] Connection header received
0620 23:13:17.875678 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.875679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.875694 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.875783 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:13:17.875897 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.876382 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.877200 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.877413 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.877498 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.877705 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:13:17.877794 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:13:17.877883 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.877939 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.877956 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":75,"threads_started":1}
I20260620 23:13:17.878053 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.875330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:37081 (local address 127.0.0.1:42134)
0620 23:13:17.875520 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.875532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.875583 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.875787 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.875788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.875789 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:13:17.875842 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.875844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.876464 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.876466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.877102 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.877105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.877501 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.877502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.877576 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.877589 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:13:17.877871 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:13:17.877880 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.877934 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.877982 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260620 23:13:17.879160 24649 raft_consensus.cc:1261] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Refusing update from remote peer 7d70399e26494d619651210472ad8718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:17.879184 24369 raft_consensus.cc:1261] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Refusing update from remote peer 7d70399e26494d619651210472ad8718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:17.879401 24768 consensus_queue.cc:1035] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:17.879537 24729 consensus_queue.cc:1035] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:17.880015 24731 tablet_replica.cc:439] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4: TxnStatusTablet state changed. Reason: New leader 7d70399e26494d619651210472ad8718. Latest consensus state: current_term: 1 leader_uuid: "7d70399e26494d619651210472ad8718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } } }
I20260620 23:13:17.880115 24731 tablet_replica.cc:442] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:13:17.880156 24730 tablet_replica.cc:439] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e: TxnStatusTablet state changed. Reason: New leader 7d70399e26494d619651210472ad8718. Latest consensus state: current_term: 1 leader_uuid: "7d70399e26494d619651210472ad8718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } } }
I20260620 23:13:17.880242 24730 tablet_replica.cc:442] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:13:17.880848 24729 tablet_replica.cc:439] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d70399e26494d619651210472ad8718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } } }
I20260620 23:13:17.880949 24729 tablet_replica.cc:442] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:13:17.880996 24729 tablet_replica.cc:439] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d70399e26494d619651210472ad8718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } } }
I20260620 23:13:17.881032 24729 tablet_replica.cc:442] T 7f02a0f5c11b4820a34f87c9975b3217 P 7d70399e26494d619651210472ad8718: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:13:17.881211 24730 tablet_replica.cc:439] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d70399e26494d619651210472ad8718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } } }
I20260620 23:13:17.881294 24730 tablet_replica.cc:442] T 7f02a0f5c11b4820a34f87c9975b3217 P b90e1afca8dd4b27a7c253fe19c8e11e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:13:17.881481 24731 tablet_replica.cc:439] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d70399e26494d619651210472ad8718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } } }
I20260620 23:13:17.881558 24731 tablet_replica.cc:442] T 7f02a0f5c11b4820a34f87c9975b3217 P 1d225484bf8545a1b856d5c7e2de01b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:13:17.886947 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.884456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:41205 (local address 127.16.79.2:37081)
0620 23:13:17.884496 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.884498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.884577 (+ 79us) server_negotiation.cc:407] Connection header received
0620 23:13:17.884650 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.884650 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.884657 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.884751 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:13:17.884858 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.885264 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.886355 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.886519 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.886580 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.886762 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:13:17.886784 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:13:17.886822 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.886890 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.886904 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:13:17.887012 24784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.884370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:37081 (local address 127.16.79.1:41205)
0620 23:13:17.884521 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.884532 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.884594 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.884755 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.884756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.884758 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:13:17.884800 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.884801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.885338 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.885339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.886264 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.886266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.886599 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.886600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.886670 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.886684 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:13:17.886837 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:13:17.886839 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.886890 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.886937 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":94,"threads_started":1}
I20260620 23:13:17.889138 24649 raft_consensus.cc:1261] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Refusing update from remote peer b90e1afca8dd4b27a7c253fe19c8e11e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:17.889277 24509 raft_consensus.cc:1261] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Refusing update from remote peer b90e1afca8dd4b27a7c253fe19c8e11e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:17.889398 24730 consensus_queue.cc:1035] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:17.889501 24743 consensus_queue.cc:1035] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:17.893559 24775 mvcc.cc:205] Tried to move back new op lower bound from 7299060522552459264 to 7299060521257517056. Current Snapshot: MvccSnapshot[applied={T|T < 7299060522552459264}]
I20260620 23:13:17.898407 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.895345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:60099 (local address 127.16.79.1:43675)
0620 23:13:17.895416 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.895417 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.895502 (+ 85us) server_negotiation.cc:407] Connection header received
0620 23:13:17.895597 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.895598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.895609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.895676 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:13:17.895832 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.896528 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.897673 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.897972 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.898047 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.898195 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:13:17.898226 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:13:17.898279 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.898325 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.898343 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:13:17.898427 24790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.895196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:43675 (local address 127.16.79.2:60099)
0620 23:13:17.895442 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:17.895453 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:17.895528 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:17.895691 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:17.895691 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:17.895692 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:13:17.895739 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.895740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.896611 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.896611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.897608 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:17.897610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.898060 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.898061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:17.898109 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.898120 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:13:17.898284 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:13:17.898286 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:13:17.898322 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 23:13:17.898353 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260620 23:13:17.903924 24210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:17.904026 24210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:17.904958 24210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1: Sending DeleteTablet for 3 replicas of tablet d4313a9d397745c78911fada6077323a
I20260620 23:13:17.905381 24489 tablet_service.cc:1514] Processing DeleteTablet for tablet d4313a9d397745c78911fada6077323a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:17 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:42130
I20260620 23:13:17.905366 24628 tablet_service.cc:1514] Processing DeleteTablet for tablet d4313a9d397745c78911fada6077323a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:17 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:46924
I20260620 23:13:17.908272 24793 tablet_replica.cc:330] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4: stopping tablet replica
I20260620 23:13:17.908242 24348 tablet_service.cc:1514] Processing DeleteTablet for tablet d4313a9d397745c78911fada6077323a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:17 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:38352
I20260620 23:13:17.908357 24793 raft_consensus.cc:2229] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:17.908401 24793 raft_consensus.cc:2258] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:17.908540 24795 tablet_replica.cc:330] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e: stopping tablet replica
I20260620 23:13:17.908655 24795 raft_consensus.cc:2229] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:13:17.908679 24793 ts_tablet_manager.cc:1905] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:17.908726 24792 tablet_replica.cc:330] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718: stopping tablet replica
I20260620 23:13:17.908777 24792 raft_consensus.cc:2229] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:17.908813 24792 raft_consensus.cc:2258] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:17.908896 24795 raft_consensus.cc:2258] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:17.909054 24792 ts_tablet_manager.cc:1905] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:17.909195 24795 ts_tablet_manager.cc:1905] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:17.910290 24793 ts_tablet_manager.cc:1918] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:13:17.910349 24793 log.cc:1196] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/d4313a9d397745c78911fada6077323a
I20260620 23:13:17.910576 24793 ts_tablet_manager.cc:1939] T d4313a9d397745c78911fada6077323a P 1d225484bf8545a1b856d5c7e2de01b4: Deleting consensus metadata
I20260620 23:13:17.910635 24792 ts_tablet_manager.cc:1918] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:13:17.910691 24792 log.cc:1196] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/d4313a9d397745c78911fada6077323a
I20260620 23:13:17.910935 24792 ts_tablet_manager.cc:1939] T d4313a9d397745c78911fada6077323a P 7d70399e26494d619651210472ad8718: Deleting consensus metadata
I20260620 23:13:17.911022 24199 catalog_manager.cc:4841] TS 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751): tablet d4313a9d397745c78911fada6077323a (table test-table [id=e2fbbd8b5f9242458089da1e273d7fca]) successfully deleted
I20260620 23:13:17.911199 24795 ts_tablet_manager.cc:1918] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:13:17.911238 24197 catalog_manager.cc:4841] TS 7d70399e26494d619651210472ad8718 (127.16.79.2:37081): tablet d4313a9d397745c78911fada6077323a (table test-table [id=e2fbbd8b5f9242458089da1e273d7fca]) successfully deleted
I20260620 23:13:17.911281 24795 log.cc:1196] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/wal/wals/d4313a9d397745c78911fada6077323a
I20260620 23:13:17.911490 24795 ts_tablet_manager.cc:1939] T d4313a9d397745c78911fada6077323a P b90e1afca8dd4b27a7c253fe19c8e11e: Deleting consensus metadata
I20260620 23:13:17.911787 24196 catalog_manager.cc:4841] TS b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675): tablet d4313a9d397745c78911fada6077323a (table test-table [id=e2fbbd8b5f9242458089da1e273d7fca]) successfully deleted
Jun 20 23:13:17 dist-test-slave-pswb krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:13:17 dist-test-slave-pswb krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:17 dist-test-slave-pswb krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:13:17.931236 24261 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:17.925241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.16.79.62:34167)
0620 23:13:17.925322 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.925324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.925393 (+ 69us) server_negotiation.cc:407] Connection header received
0620 23:13:17.925471 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.925472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.925480 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.925536 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:17.926246 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.926741 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.927777 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.927957 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.928064 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.931174 (+ 3110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54452: BlockingRecv error: recv got EOF from 127.0.0.1:54452 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 23:13:17 dist-test-slave-pswb krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM
I20260620 23:13:17.939607 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:17.932223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.16.79.62:34167)
0620 23:13:17.932271 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:17.932273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:17.932338 (+ 65us) server_negotiation.cc:407] Connection header received
0620 23:13:17.932385 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:17.932386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:17.932396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:17.932440 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:17.933312 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.933826 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.934699 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:17.934900 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:17.934992 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:17.937120 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:17.937121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:17.937122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:17.937128 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:17.938509 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.938857 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.938858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.938859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.938917 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:17.939124 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:17.939124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:17.939125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:17.939236 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:17.939340 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:17.939440 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 23:13:17.939513 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:13:17.941195 24210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:13:17.941360 24210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:13:17.942660 24348 tablet_service.cc:1467] Processing CreateTablet for tablet d8c803b5604d496e841a2c5ae2864f3e (DEFAULT_TABLE table=test-table [id=6a04a2b53db4463093e76a93dc5ef04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:17.942737 24628 tablet_service.cc:1467] Processing CreateTablet for tablet d8c803b5604d496e841a2c5ae2864f3e (DEFAULT_TABLE table=test-table [id=6a04a2b53db4463093e76a93dc5ef04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:17.942785 24348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c803b5604d496e841a2c5ae2864f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:17.942740 24489 tablet_service.cc:1467] Processing CreateTablet for tablet d8c803b5604d496e841a2c5ae2864f3e (DEFAULT_TABLE table=test-table [id=6a04a2b53db4463093e76a93dc5ef04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:17.942833 24628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c803b5604d496e841a2c5ae2864f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:17.942893 24489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c803b5604d496e841a2c5ae2864f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:17.944057 24724 tablet_bootstrap.cc:492] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e: Bootstrap starting.
I20260620 23:13:17.944204 24727 tablet_bootstrap.cc:492] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4: Bootstrap starting.
I20260620 23:13:17.944432 24724 tablet_bootstrap.cc:654] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:17.944856 24727 tablet_bootstrap.cc:654] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:17.945098 24724 tablet_bootstrap.cc:492] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e: No bootstrap required, opened a new log
I20260620 23:13:17.945168 24724 ts_tablet_manager.cc:1397] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:17.945178 24723 tablet_bootstrap.cc:492] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718: Bootstrap starting.
I20260620 23:13:17.945276 24724 raft_consensus.cc:348] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.945362 24724 raft_consensus.cc:374] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:17.945389 24724 raft_consensus.cc:729] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90e1afca8dd4b27a7c253fe19c8e11e, State: Initialized, Role: FOLLOWER
I20260620 23:13:17.945428 24724 consensus_queue.cc:260] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.945446 24723 tablet_bootstrap.cc:654] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:17.945523 24724 ts_tablet_manager.cc:1428] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:17.946095 24723 tablet_bootstrap.cc:492] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718: No bootstrap required, opened a new log
I20260620 23:13:17.946118 24727 tablet_bootstrap.cc:492] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4: No bootstrap required, opened a new log
I20260620 23:13:17.946168 24727 ts_tablet_manager.cc:1397] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 23:13:17.946170 24723 ts_tablet_manager.cc:1397] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:17.946275 24723 raft_consensus.cc:348] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.946309 24727 raft_consensus.cc:348] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.946359 24723 raft_consensus.cc:374] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:17.946372 24727 raft_consensus.cc:374] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:17.946389 24723 raft_consensus.cc:729] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d70399e26494d619651210472ad8718, State: Initialized, Role: FOLLOWER
I20260620 23:13:17.946395 24727 raft_consensus.cc:729] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d225484bf8545a1b856d5c7e2de01b4, State: Initialized, Role: FOLLOWER
I20260620 23:13:17.946429 24723 consensus_queue.cc:260] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.946437 24727 consensus_queue.cc:260] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:17.946519 24723 ts_tablet_manager.cc:1428] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:17.946539 24727 ts_tablet_manager.cc:1428] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:18.012946 24731 raft_consensus.cc:482] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:18.013089 24731 raft_consensus.cc:504] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:18.013310 24731 leader_election.cc:288] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675), 7d70399e26494d619651210472ad8718 (127.16.79.2:37081)
I20260620 23:13:18.013494 24369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c803b5604d496e841a2c5ae2864f3e" candidate_uuid: "1d225484bf8545a1b856d5c7e2de01b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" is_pre_election: true
I20260620 23:13:18.013579 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c803b5604d496e841a2c5ae2864f3e" candidate_uuid: "1d225484bf8545a1b856d5c7e2de01b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d70399e26494d619651210472ad8718" is_pre_election: true
I20260620 23:13:18.013630 24369 raft_consensus.cc:2454] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d225484bf8545a1b856d5c7e2de01b4 in term 0.
I20260620 23:13:18.013690 24509 raft_consensus.cc:2454] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d225484bf8545a1b856d5c7e2de01b4 in term 0.
I20260620 23:13:18.013823 24580 leader_election.cc:302] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d225484bf8545a1b856d5c7e2de01b4, b90e1afca8dd4b27a7c253fe19c8e11e; no voters:
I20260620 23:13:18.013947 24731 raft_consensus.cc:2781] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:13:18.014003 24731 raft_consensus.cc:482] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:18.014024 24731 raft_consensus.cc:3037] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:18.014923 24731 raft_consensus.cc:504] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:18.015067 24731 leader_election.cc:288] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [CANDIDATE]: Term 1 election: Requested vote from peers b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675), 7d70399e26494d619651210472ad8718 (127.16.79.2:37081)
I20260620 23:13:18.015240 24369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c803b5604d496e841a2c5ae2864f3e" candidate_uuid: "1d225484bf8545a1b856d5c7e2de01b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e"
I20260620 23:13:18.015276 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c803b5604d496e841a2c5ae2864f3e" candidate_uuid: "1d225484bf8545a1b856d5c7e2de01b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d70399e26494d619651210472ad8718"
I20260620 23:13:18.015336 24369 raft_consensus.cc:3037] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:18.015347 24509 raft_consensus.cc:3037] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:18.015941 24509 raft_consensus.cc:2454] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d225484bf8545a1b856d5c7e2de01b4 in term 1.
I20260620 23:13:18.015944 24369 raft_consensus.cc:2454] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d225484bf8545a1b856d5c7e2de01b4 in term 1.
I20260620 23:13:18.016103 24581 leader_election.cc:302] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d225484bf8545a1b856d5c7e2de01b4, 7d70399e26494d619651210472ad8718; no voters:
I20260620 23:13:18.016187 24731 raft_consensus.cc:2781] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:18.016358 24731 raft_consensus.cc:686] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 LEADER]: Becoming Leader. State: Replica: 1d225484bf8545a1b856d5c7e2de01b4, State: Running, Role: LEADER
I20260620 23:13:18.016450 24731 consensus_queue.cc:237] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:18.017328 24211 catalog_manager.cc:5495] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3). New cstate: current_term: 1 leader_uuid: "1d225484bf8545a1b856d5c7e2de01b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } health_report { overall_health: HEALTHY } } }
I20260620 23:13:18.041419 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:18.038834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.16.79.3:34751)
0620 23:13:18.038901 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:18.038903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:18.038909 (+ 6us) server_negotiation.cc:407] Connection header received
0620 23:13:18.038938 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:18.038939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:18.038957 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:18.039034 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:13:18.039198 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:18.039677 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:18.040435 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:18.040625 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:18.040879 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:18.041069 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:13:18.041174 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:13:18.041272 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:18.041315 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 23:13:18.041335 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:13:18.042413 24509 raft_consensus.cc:1261] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Refusing update from remote peer 1d225484bf8545a1b856d5c7e2de01b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:18.042413 24369 raft_consensus.cc:1261] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Refusing update from remote peer 1d225484bf8545a1b856d5c7e2de01b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:18.042630 24810 consensus_queue.cc:1035] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:18.042758 24731 consensus_queue.cc:1035] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:18.045859 24210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:18.045956 24210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:18.046563 24210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1: Sending DeleteTablet for 3 replicas of tablet d8c803b5604d496e841a2c5ae2864f3e
I20260620 23:13:18.046890 24348 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c803b5604d496e841a2c5ae2864f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:18 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:38352
I20260620 23:13:18.046912 24628 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c803b5604d496e841a2c5ae2864f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:18 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:46924
I20260620 23:13:18.046983 24795 tablet_replica.cc:330] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e: stopping tablet replica
I20260620 23:13:18.047030 24795 raft_consensus.cc:2229] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:18.047068 24795 raft_consensus.cc:2258] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:18.046993 24210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:13:18.047132 24210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:13:18.047135 24793 tablet_replica.cc:330] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4: stopping tablet replica
I20260620 23:13:18.047184 24793 raft_consensus.cc:2229] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:13:18.047248 24793 raft_consensus.cc:2258] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:18.047479 24489 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c803b5604d496e841a2c5ae2864f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:18 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:42130
I20260620 23:13:18.047582 24792 tablet_replica.cc:330] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718: stopping tablet replica
I20260620 23:13:18.047638 24792 raft_consensus.cc:2229] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:18.047688 24792 raft_consensus.cc:2258] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:18.047809 24795 ts_tablet_manager.cc:1905] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:18.047943 24792 ts_tablet_manager.cc:1905] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:18.049643 24792 ts_tablet_manager.cc:1918] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:18.049705 24792 log.cc:1196] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/d8c803b5604d496e841a2c5ae2864f3e
I20260620 23:13:18.049921 24792 ts_tablet_manager.cc:1939] T d8c803b5604d496e841a2c5ae2864f3e P 7d70399e26494d619651210472ad8718: Deleting consensus metadata
I20260620 23:13:18.050199 24197 catalog_manager.cc:4841] TS 7d70399e26494d619651210472ad8718 (127.16.79.2:37081): tablet d8c803b5604d496e841a2c5ae2864f3e (table test-table [id=6a04a2b53db4463093e76a93dc5ef04d]) successfully deleted
I20260620 23:13:18.050494 24348 tablet_service.cc:1467] Processing CreateTablet for tablet 38f86a2207544d5e833e20bb1613aae6 (DEFAULT_TABLE table=test-table [id=5e3146bbf85543b59f2d06820ef55d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:18.050506 24628 tablet_service.cc:1467] Processing CreateTablet for tablet 38f86a2207544d5e833e20bb1613aae6 (DEFAULT_TABLE table=test-table [id=5e3146bbf85543b59f2d06820ef55d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:18.050613 24628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f86a2207544d5e833e20bb1613aae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:18.050712 24489 tablet_service.cc:1467] Processing CreateTablet for tablet 38f86a2207544d5e833e20bb1613aae6 (DEFAULT_TABLE table=test-table [id=5e3146bbf85543b59f2d06820ef55d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:18.050796 24489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f86a2207544d5e833e20bb1613aae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:18.051548 24723 tablet_bootstrap.cc:492] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718: Bootstrap starting.
I20260620 23:13:18.051857 24727 tablet_bootstrap.cc:492] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4: Bootstrap starting.
I20260620 23:13:18.051890 24723 tablet_bootstrap.cc:654] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:18.052222 24727 tablet_bootstrap.cc:654] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:18.052724 24795 ts_tablet_manager.cc:1918] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:18.052784 24795 log.cc:1196] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-0/wal/wals/d8c803b5604d496e841a2c5ae2864f3e
I20260620 23:13:18.052871 24727 tablet_bootstrap.cc:492] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4: No bootstrap required, opened a new log
I20260620 23:13:18.052912 24727 ts_tablet_manager.cc:1397] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:18.053002 24795 ts_tablet_manager.cc:1939] T d8c803b5604d496e841a2c5ae2864f3e P b90e1afca8dd4b27a7c253fe19c8e11e: Deleting consensus metadata
I20260620 23:13:18.053062 24727 raft_consensus.cc:348] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:18.053117 24727 raft_consensus.cc:374] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:18.053122 24348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f86a2207544d5e833e20bb1613aae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:18.053138 24727 raft_consensus.cc:729] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d225484bf8545a1b856d5c7e2de01b4, State: Initialized, Role: FOLLOWER
I20260620 23:13:18.053179 24727 consensus_queue.cc:260] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:18.053292 24727 ts_tablet_manager.cc:1428] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:18.053351 24196 catalog_manager.cc:4841] TS b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675): tablet d8c803b5604d496e841a2c5ae2864f3e (table test-table [id=6a04a2b53db4463093e76a93dc5ef04d]) successfully deleted
I20260620 23:13:18.054164 24724 tablet_bootstrap.cc:492] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e: Bootstrap starting.
I20260620 23:13:18.054471 24724 tablet_bootstrap.cc:654] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:18.055039 24793 ts_tablet_manager.cc:1905] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:18.055136 24724 tablet_bootstrap.cc:492] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e: No bootstrap required, opened a new log
I20260620 23:13:18.055186 24724 ts_tablet_manager.cc:1397] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:18.055280 24724 raft_consensus.cc:348] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:18.055342 24724 raft_consensus.cc:374] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:18.055361 24724 raft_consensus.cc:729] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90e1afca8dd4b27a7c253fe19c8e11e, State: Initialized, Role: FOLLOWER
I20260620 23:13:18.055411 24724 consensus_queue.cc:260] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:18.055490 24724 ts_tablet_manager.cc:1428] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:18.056571 24793 ts_tablet_manager.cc:1918] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:18.056631 24793 log.cc:1196] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/d8c803b5604d496e841a2c5ae2864f3e
I20260620 23:13:18.056856 24793 ts_tablet_manager.cc:1939] T d8c803b5604d496e841a2c5ae2864f3e P 1d225484bf8545a1b856d5c7e2de01b4: Deleting consensus metadata
I20260620 23:13:18.057107 24199 catalog_manager.cc:4841] TS 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751): tablet d8c803b5604d496e841a2c5ae2864f3e (table test-table [id=6a04a2b53db4463093e76a93dc5ef04d]) successfully deleted
I20260620 23:13:18.057525 24723 tablet_bootstrap.cc:492] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718: No bootstrap required, opened a new log
I20260620 23:13:18.057567 24723 ts_tablet_manager.cc:1397] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 23:13:18.057675 24723 raft_consensus.cc:348] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:18.057745 24723 raft_consensus.cc:374] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:18.057765 24723 raft_consensus.cc:729] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d70399e26494d619651210472ad8718, State: Initialized, Role: FOLLOWER
I20260620 23:13:18.057811 24723 consensus_queue.cc:260] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:18.057925 24723 ts_tablet_manager.cc:1428] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:18.072808 24768 raft_consensus.cc:482] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:18.072875 24768 raft_consensus.cc:504] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:18.073014 24768 leader_election.cc:288] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675), 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751)
I20260620 23:13:18.073154 24369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f86a2207544d5e833e20bb1613aae6" candidate_uuid: "7d70399e26494d619651210472ad8718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" is_pre_election: true
I20260620 23:13:18.073194 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f86a2207544d5e833e20bb1613aae6" candidate_uuid: "7d70399e26494d619651210472ad8718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d225484bf8545a1b856d5c7e2de01b4" is_pre_election: true
I20260620 23:13:18.073235 24369 raft_consensus.cc:2454] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d70399e26494d619651210472ad8718 in term 0.
I20260620 23:13:18.073261 24649 raft_consensus.cc:2454] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d70399e26494d619651210472ad8718 in term 0.
I20260620 23:13:18.073405 24440 leader_election.cc:302] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d70399e26494d619651210472ad8718, b90e1afca8dd4b27a7c253fe19c8e11e; no voters:
I20260620 23:13:18.073500 24768 raft_consensus.cc:2781] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:13:18.073539 24768 raft_consensus.cc:482] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:18.073570 24768 raft_consensus.cc:3037] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:18.074093 24768 raft_consensus.cc:504] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:18.074225 24768 leader_election.cc:288] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [CANDIDATE]: Term 1 election: Requested vote from peers b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675), 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751)
I20260620 23:13:18.074378 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f86a2207544d5e833e20bb1613aae6" candidate_uuid: "7d70399e26494d619651210472ad8718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d225484bf8545a1b856d5c7e2de01b4"
I20260620 23:13:18.074470 24649 raft_consensus.cc:3037] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:18.074471 24369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f86a2207544d5e833e20bb1613aae6" candidate_uuid: "7d70399e26494d619651210472ad8718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e"
I20260620 23:13:18.074543 24369 raft_consensus.cc:3037] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:18.075136 24649 raft_consensus.cc:2454] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d70399e26494d619651210472ad8718 in term 1.
I20260620 23:13:18.075269 24369 raft_consensus.cc:2454] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d70399e26494d619651210472ad8718 in term 1.
I20260620 23:13:18.075311 24443 leader_election.cc:302] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d225484bf8545a1b856d5c7e2de01b4, 7d70399e26494d619651210472ad8718; no voters:
I20260620 23:13:18.075541 24768 raft_consensus.cc:2781] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:18.075601 24768 raft_consensus.cc:686] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 1 LEADER]: Becoming Leader. State: Replica: 7d70399e26494d619651210472ad8718, State: Running, Role: LEADER
I20260620 23:13:18.075649 24768 consensus_queue.cc:237] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } }
I20260620 23:13:18.076177 24210 catalog_manager.cc:5495] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d70399e26494d619651210472ad8718 (127.16.79.2). New cstate: current_term: 1 leader_uuid: "7d70399e26494d619651210472ad8718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d70399e26494d619651210472ad8718" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 37081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 } health_report { overall_health: UNKNOWN } } }
I20260620 23:13:18.088515 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:18.086316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.16.79.2:37081)
0620 23:13:18.086358 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:18.086360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:18.086369 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:18.086440 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:18.086441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:18.086452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:18.086505 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:13:18.086612 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:18.087094 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:18.087836 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:18.088031 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:18.088106 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:18.088256 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:13:18.088293 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:13:18.088354 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:18.088414 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 23:13:18.088433 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:13:18.092597 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:18.090554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:39383 (local address 127.16.79.2:37081)
0620 23:13:18.090586 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:18.090588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:18.090604 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:13:18.090622 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:18.090623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:18.090631 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:18.090687 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:13:18.090778 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:18.091216 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:18.091956 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:18.092151 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:18.092231 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:18.092368 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:13:18.092397 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:13:18.092442 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:18.092494 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 23:13:18.092529 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":18}
I20260620 23:13:18.092640 24790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:18.090484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:37081 (local address 127.16.79.2:39383)
0620 23:13:18.090534 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:18.090543 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:18.090580 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:18.090690 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:18.090690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:18.090691 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:13:18.090723 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:18.090725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:18.091293 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:18.091294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:18.091860 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:18.091861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:18.092232 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:18.092232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:18.092279 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:18.092284 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:13:18.092440 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:13:18.092442 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:13:18.092491 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 23:13:18.092548 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":1}
I20260620 23:13:18.095304 24649 raft_consensus.cc:1261] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Refusing update from remote peer 7d70399e26494d619651210472ad8718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:18.095387 24369 raft_consensus.cc:1261] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Refusing update from remote peer 7d70399e26494d619651210472ad8718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:18.095530 24822 consensus_queue.cc:1035] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d225484bf8545a1b856d5c7e2de01b4" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:18.095609 24729 consensus_queue.cc:1035] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90e1afca8dd4b27a7c253fe19c8e11e" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:18.105553 24211 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:18.105660 24211 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:18.106228 24211 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcbab62969e544d590897860477f52a1: Sending DeleteTablet for 3 replicas of tablet 38f86a2207544d5e833e20bb1613aae6
I20260620 23:13:18.106434 24489 tablet_service.cc:1514] Processing DeleteTablet for tablet 38f86a2207544d5e833e20bb1613aae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:18 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:42130
I20260620 23:13:18.106483 24628 tablet_service.cc:1514] Processing DeleteTablet for tablet 38f86a2207544d5e833e20bb1613aae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:18 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:46924
I20260620 23:13:18.106515 24792 tablet_replica.cc:330] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718: stopping tablet replica
I20260620 23:13:18.106551 24793 tablet_replica.cc:330] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4: stopping tablet replica
I20260620 23:13:18.106564 24792 raft_consensus.cc:2229] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:13:18.106590 24793 raft_consensus.cc:2229] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:18.106617 24793 raft_consensus.cc:2258] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:18.106647 24792 raft_consensus.cc:2258] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:18.106922 24348 tablet_service.cc:1514] Processing DeleteTablet for tablet 38f86a2207544d5e833e20bb1613aae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:18 UTC) from {username='oryx', principal='oryx/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:38352
I20260620 23:13:18.107012 24795 tablet_replica.cc:330] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e: stopping tablet replica
I20260620 23:13:18.107066 24795 raft_consensus.cc:2229] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:18.107103 24795 raft_consensus.cc:2258] T 38f86a2207544d5e833e20bb1613aae6 P b90e1afca8dd4b27a7c253fe19c8e11e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:18.107288 24792 ts_tablet_manager.cc:1905] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:18.108532 24792 ts_tablet_manager.cc:1918] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:13:18.108601 24792 log.cc:1196] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/38f86a2207544d5e833e20bb1613aae6
I20260620 23:13:18.108857 24792 ts_tablet_manager.cc:1939] T 38f86a2207544d5e833e20bb1613aae6 P 7d70399e26494d619651210472ad8718: Deleting consensus metadata
I20260620 23:13:18.109016 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 24284
I20260620 23:13:18.110741 24793 ts_tablet_manager.cc:1905] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:18.110757 24197 catalog_manager.cc:4841] TS 7d70399e26494d619651210472ad8718 (127.16.79.2:37081): tablet 38f86a2207544d5e833e20bb1613aae6 (table test-table [id=5e3146bbf85543b59f2d06820ef55d5f]) successfully deleted
I20260620 23:13:18.112179 24793 ts_tablet_manager.cc:1918] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:13:18.112251 24793 log.cc:1196] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/38f86a2207544d5e833e20bb1613aae6
I20260620 23:13:18.112478 24793 ts_tablet_manager.cc:1939] T 38f86a2207544d5e833e20bb1613aae6 P 1d225484bf8545a1b856d5c7e2de01b4: Deleting consensus metadata
I20260620 23:13:18.112792 24199 catalog_manager.cc:4841] TS 1d225484bf8545a1b856d5c7e2de01b4 (127.16.79.3:34751): tablet 38f86a2207544d5e833e20bb1613aae6 (table test-table [id=5e3146bbf85543b59f2d06820ef55d5f]) successfully deleted
W20260620 23:13:18.117308 24196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:43675 (error 108)
I20260620 23:13:18.117564 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 24424
I20260620 23:13:18.121450 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:18.117734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:43675 (local address 127.0.0.1:38368)
0620 23:13:18.121355 (+ 3621us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:18.121403 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:43675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3602,"thread_start_us":33,"threads_started":1}
W20260620 23:13:18.121541 24196 catalog_manager.cc:4584] TS b90e1afca8dd4b27a7c253fe19c8e11e (127.16.79.1:43675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38f86a2207544d5e833e20bb1613aae6: Network error: Client connection negotiation failed: client connection to 127.16.79.1:43675: connect: Connection refused (error 111)
I20260620 23:13:18.124625 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 24564
I20260620 23:13:18.129316 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 24180
2026-06-20T23:13:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2754 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:13:18 dist-test-slave-pswb krb5kdc[24829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:13:18 dist-test-slave-pswb krb5kdc[24829](info): set up 2 sockets
Jun 20 23:13:18 dist-test-slave-pswb krb5kdc[24829](info): commencing operation
krb5kdc: starting...
W20260620 23:13:19.177320 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:13:19 dist-test-slave-pswb krb5kdc[24829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:13:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:13:19.296541 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:38837
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:38837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:19.372941 24842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:19.373126 24842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:19.373147 24842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:19.374183 24842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:19.374221 24842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:19.374235 24842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:19.374248 24842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:19.374260 24842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:19.375604 24842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:38837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:38837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:19.375774 24842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:19.375941 24842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:19.380673 24848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:19.380707 24847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:19.380681 24850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:19.380838 24842 server_base.cc:1034] running on GCE node
I20260620 23:13:19.381134 24842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:19.381430 24842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:19.382598 24842 hybrid_clock.cc:648] HybridClock initialized: now 1781997199382587 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T23:13:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1253 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:13:19 dist-test-slave-pswb krb5kdc[24855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:13:19 dist-test-slave-pswb krb5kdc[24855](info): set up 2 sockets
Jun 20 23:13:19 dist-test-slave-pswb krb5kdc[24855](info): commencing operation
krb5kdc: starting...
W20260620 23:13:20.429299 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:13:20 dist-test-slave-pswb krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:13:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:13:20.543615 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45187
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:45187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:20.623893 24868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:20.624106 24868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:20.624140 24868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:20.625301 24868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:20.625365 24868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:20.625391 24868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:20.625423 24868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:20.625448 24868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:20.626893 24868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:45187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:20.627153 24868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:20.627369 24868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:20.631903 24873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:20.631932 24874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:20.632009 24876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:20.632329 24868 server_base.cc:1034] running on GCE node
I20260620 23:13:20.632505 24868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:20.632763 24868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:20.633909 24868 hybrid_clock.cc:648] HybridClock initialized: now 1781997200633888 us; error 37 us; skew 500 ppm
Jun 20 23:13:20 dist-test-slave-pswb krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:20.640993 24868 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:13:20.641361 24868 webserver.cc:460] Webserver started at http://127.16.79.62:34527/ using document root <none> and password file <none>
I20260620 23:13:20.641539 24868 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:20.641580 24868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:20.641661 24868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:20.642560 24868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "0479abcb05c34fe4b2e06ad404b6cb39"
format_stamp: "Formatted at 2026-06-20 23:13:20 on dist-test-slave-pswb"
server_key: "4e3a3e029ba396cb504b2cfd9ad47713"
server_key_iv: "2c5f4413ed113c9f14a8b239fb052188"
server_key_version: "encryptionkey@0"
I20260620 23:13:20.642839 24868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "0479abcb05c34fe4b2e06ad404b6cb39"
format_stamp: "Formatted at 2026-06-20 23:13:20 on dist-test-slave-pswb"
server_key: "4e3a3e029ba396cb504b2cfd9ad47713"
server_key_iv: "2c5f4413ed113c9f14a8b239fb052188"
server_key_version: "encryptionkey@0"
I20260620 23:13:20.644045 24868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 23:13:20.644801 24883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:20.644980 24868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:20.645078 24868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "0479abcb05c34fe4b2e06ad404b6cb39"
format_stamp: "Formatted at 2026-06-20 23:13:20 on dist-test-slave-pswb"
server_key: "4e3a3e029ba396cb504b2cfd9ad47713"
server_key_iv: "2c5f4413ed113c9f14a8b239fb052188"
server_key_version: "encryptionkey@0"
I20260620 23:13:20.645159 24868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:20.653925 24868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:20.656092 24868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:20.656240 24868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:20.659996 24868 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:45187
I20260620 23:13:20.660027 24935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:45187 every 8 connection(s)
I20260620 23:13:20.660328 24868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:20.661156 24936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:20.663162 24936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39: Bootstrap starting.
I20260620 23:13:20.663695 24936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:20.663967 24936 log.cc:824] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:20.664594 24936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39: No bootstrap required, opened a new log
I20260620 23:13:20.666299 24936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45187 } }
I20260620 23:13:20.666424 24936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:20.666446 24936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0479abcb05c34fe4b2e06ad404b6cb39, State: Initialized, Role: FOLLOWER
I20260620 23:13:20.666594 24936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45187 } }
I20260620 23:13:20.666661 24936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:20.666687 24936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:20.666728 24936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:20.667349 24936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45187 } }
I20260620 23:13:20.667459 24936 leader_election.cc:302] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0479abcb05c34fe4b2e06ad404b6cb39; no voters:
I20260620 23:13:20.667634 24936 leader_election.cc:288] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:20.667701 24939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:20.667871 24939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 1 LEADER]: Becoming Leader. State: Replica: 0479abcb05c34fe4b2e06ad404b6cb39, State: Running, Role: LEADER
I20260620 23:13:20.667874 24936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:20.667999 24939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45187 } }
I20260620 23:13:20.668341 24940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45187 } } }
I20260620 23:13:20.668519 24940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:20.668522 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 24868
I20260620 23:13:20.668622 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:20.668865 16700 external_mini_cluster.cc:1428] Setting key 64101428b189bce17a6106d7b0fe5d39
I20260620 23:13:20.668893 24945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:20.668341 24941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0479abcb05c34fe4b2e06ad404b6cb39. Latest consensus state: current_term: 1 leader_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45187 } } }
I20260620 23:13:20.669155 24941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:20.669466 24945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:20.671033 24945 catalog_manager.cc:1347] Generated new cluster ID: e3ed0549ed254bd7a2061e6af349b09a
I20260620 23:13:20.671084 24945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:13:20 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:20.677122 24945 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:20.677731 24945 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:20.680513 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:20.669482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.16.79.62:45187)
0620 23:13:20.669853 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:20.669861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:20.669898 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:13:20.670072 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:20.670077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:20.670147 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:20.670278 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:20.671040 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:20.672408 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:20.673514 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:20.674114 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:20.674206 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:20.676987 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:20.676990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:20.676991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:20.677002 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:20.678943 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:20.679323 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:20.679327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:20.679327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:20.679432 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:20.679631 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:20.679632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:20.679632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:20.679922 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:20.680078 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:20.680175 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 23:13:20.680313 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":323,"thread_start_us":60,"threads_started":1}
I20260620 23:13:20.696321 24945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39: Generated new TSK 0
I20260620 23:13:20.696514 24945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:13:20.727737 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45187
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:20.806612 24962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:20.806833 24962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:20.806869 24962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:20.807994 24962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:20.808065 24962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:20.808120 24962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:20.809561 24962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:20.809813 24962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:20.810034 24962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:20.814857 24968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:20.814872 24970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:20.814873 24967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:20.815196 24962 server_base.cc:1034] running on GCE node
I20260620 23:13:20.815423 24962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:20.815649 24962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:20.816764 24962 hybrid_clock.cc:648] HybridClock initialized: now 1781997200816740 us; error 38 us; skew 500 ppm
Jun 20 23:13:20 dist-test-slave-pswb krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:20.819154 24962 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:13:20.819525 24962 webserver.cc:460] Webserver started at http://127.16.79.1:42379/ using document root <none> and password file <none>
I20260620 23:13:20.819689 24962 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:20.819727 24962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:20.819810 24962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:20.820816 24962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "6490afa2333b44b7b9e7c3b4b27dfdce"
format_stamp: "Formatted at 2026-06-20 23:13:20 on dist-test-slave-pswb"
server_key: "2ebc05d041d85f2ba4e9399fbc289688"
server_key_iv: "4d7354f4ea6085b8a91ed2803296412f"
server_key_version: "encryptionkey@0"
I20260620 23:13:20.821105 24962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "6490afa2333b44b7b9e7c3b4b27dfdce"
format_stamp: "Formatted at 2026-06-20 23:13:20 on dist-test-slave-pswb"
server_key: "2ebc05d041d85f2ba4e9399fbc289688"
server_key_iv: "4d7354f4ea6085b8a91ed2803296412f"
server_key_version: "encryptionkey@0"
I20260620 23:13:20.822309 24962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:20.822932 24977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:20.823124 24962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:20.823190 24962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "6490afa2333b44b7b9e7c3b4b27dfdce"
format_stamp: "Formatted at 2026-06-20 23:13:20 on dist-test-slave-pswb"
server_key: "2ebc05d041d85f2ba4e9399fbc289688"
server_key_iv: "4d7354f4ea6085b8a91ed2803296412f"
server_key_version: "encryptionkey@0"
I20260620 23:13:20.823293 24962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:20.834649 24962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:20.836788 24962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:20.836917 24962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:20.837131 24962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:20.837421 24962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:20.837474 24962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:20.837512 24962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:20.837541 24962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:20.843214 24962 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:43313
I20260620 23:13:20.843276 25090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:43313 every 8 connection(s)
I20260620 23:13:20.843546 24962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:13:20 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:20.852257 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 24962
I20260620 23:13:20.852396 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:13:20.852587 16700 external_mini_cluster.cc:1428] Setting key 04962ffa6bf275018ec313b59602bca2
I20260620 23:13:20.852818 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:20.844178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:35535 (local address 127.16.79.62:45187)
0620 23:13:20.844250 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:20.844252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:20.844467 (+ 215us) server_negotiation.cc:407] Connection header received
0620 23:13:20.845262 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:20.845263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:20.845274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:20.845355 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:20.846331 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:20.846838 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:20.847879 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:20.848093 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:20.848166 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:20.850164 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:20.850165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:20.850165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:20.850173 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:20.851367 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:20.851768 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:20.851769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:20.851769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:20.851830 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:20.852096 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:20.852097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:20.852097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:20.852211 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:20.852290 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:20.852594 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 23:13:20.852680 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:13:20.852941 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:20.844198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:35535)
0620 23:13:20.844356 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:20.844386 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:20.845149 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:20.845423 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:20.845427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:20.846059 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:20.846228 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:20.846231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:20.846940 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:20.846941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:20.847782 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:20.847784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:20.848171 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:20.848173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:20.848313 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:20.848809 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:20.848825 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:20.850070 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:20.851462 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:20.851468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:20.851469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:20.851673 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:20.851908 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:20.851908 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:20.851909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:20.852014 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:20.852280 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:20.852286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:20.852449 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 23:13:20.852573 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 23:13:20.852731 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260620 23:13:20.853399 25091 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45187
I20260620 23:13:20.853478 25091 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:20.853655 25091 heartbeater.cc:511] Master 127.16.79.62:45187 requested a full tablet report, sending...
I20260620 23:13:20.854138 24899 ts_manager.cc:194] Registered new tserver with Master: 6490afa2333b44b7b9e7c3b4b27dfdce (127.16.79.1:43313)
I20260620 23:13:20.854949 24899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:35535
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:13:20.875540 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45187
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:20.953437 25096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:20.953639 25096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:20.953673 25096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:20.954771 25096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:20.954841 25096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:20.954888 25096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:20.956254 25096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:20.956504 25096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:20.956728 25096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:20.961418 25104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:20.961426 25101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:20.961560 25096 server_base.cc:1034] running on GCE node
W20260620 23:13:20.961476 25102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:20.962077 25096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:20.962270 25096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:20.963400 25096 hybrid_clock.cc:648] HybridClock initialized: now 1781997200963381 us; error 37 us; skew 500 ppm
Jun 20 23:13:20 dist-test-slave-pswb krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:20.965821 25096 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:13:20.966338 25096 webserver.cc:460] Webserver started at http://127.16.79.2:36381/ using document root <none> and password file <none>
I20260620 23:13:20.966585 25096 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:20.966652 25096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:20.966784 25096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:20.968036 25096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "d3263e24159c412cb2c6eccc83cc7fc2"
format_stamp: "Formatted at 2026-06-20 23:13:20 on dist-test-slave-pswb"
server_key: "2f8840a4a4e0344e5d7fc9a683bc16b2"
server_key_iv: "910730a88884049a40677c040cfd62c3"
server_key_version: "encryptionkey@0"
I20260620 23:13:20.968470 25096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "d3263e24159c412cb2c6eccc83cc7fc2"
format_stamp: "Formatted at 2026-06-20 23:13:20 on dist-test-slave-pswb"
server_key: "2f8840a4a4e0344e5d7fc9a683bc16b2"
server_key_iv: "910730a88884049a40677c040cfd62c3"
server_key_version: "encryptionkey@0"
I20260620 23:13:20.970050 25096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260620 23:13:20.970846 25111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:20.971061 25096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:20.971123 25096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "d3263e24159c412cb2c6eccc83cc7fc2"
format_stamp: "Formatted at 2026-06-20 23:13:20 on dist-test-slave-pswb"
server_key: "2f8840a4a4e0344e5d7fc9a683bc16b2"
server_key_iv: "910730a88884049a40677c040cfd62c3"
server_key_version: "encryptionkey@0"
I20260620 23:13:20.971204 25096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:20.991513 25096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:20.994092 25096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:20.994298 25096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:20.994594 25096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:20.995023 25096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:20.995090 25096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:20.995131 25096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:20.995162 25096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.001895 25096 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:38351
I20260620 23:13:21.001936 25224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:38351 every 8 connection(s)
I20260620 23:13:21.002233 25096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:21.009632 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 25096
I20260620 23:13:21.009775 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:21.009958 16700 external_mini_cluster.cc:1428] Setting key 05a26a8e8eca1e647755e38ca9963c98
I20260620 23:13:21.011441 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.002933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:54079 (local address 127.16.79.62:45187)
0620 23:13:21.003017 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:21.003020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:21.003204 (+ 184us) server_negotiation.cc:407] Connection header received
0620 23:13:21.003694 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:21.003695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:21.003707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:21.003753 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:21.004708 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.005200 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.006193 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.006386 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.006473 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.008530 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:21.008532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:21.008532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:21.008538 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:21.009957 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:21.010449 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:21.010450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:21.010451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:21.010510 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:21.010781 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:21.010782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:21.010783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:21.010901 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:21.010978 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:21.011271 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 23:13:21.011354 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:13:21.011597 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.002820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:54079)
0620 23:13:21.003051 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:21.003109 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:21.003587 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:21.003835 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:21.003838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:21.004409 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:21.004584 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.004588 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.005312 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.005313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.006087 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.006089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.006482 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.006483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.006609 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.007062 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:21.007069 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:21.008429 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:21.010058 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:21.010064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:21.010066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:21.010371 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:21.010597 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:21.010598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:21.010598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:21.010723 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:21.010970 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:21.010976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:21.011109 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 23:13:21.011231 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 23:13:21.011378 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":108,"threads_started":1}
I20260620 23:13:21.012071 25225 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45187
I20260620 23:13:21.012172 25225 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:21.012359 25225 heartbeater.cc:511] Master 127.16.79.62:45187 requested a full tablet report, sending...
I20260620 23:13:21.012821 24899 ts_manager.cc:194] Registered new tserver with Master: d3263e24159c412cb2c6eccc83cc7fc2 (127.16.79.2:38351)
I20260620 23:13:21.013207 24899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:54079
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:13:21.033813 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45187
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:21.112099 25230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:21.112291 25230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:21.112314 25230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:21.113457 25230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:21.113499 25230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:21.113526 25230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:21.114929 25230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:21.115175 25230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:21.115388 25230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:21.120316 25235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:21.120517 25230 server_base.cc:1034] running on GCE node
W20260620 23:13:21.120333 25236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:21.120333 25238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:21.121044 25230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:21.121310 25230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:21.122471 25230 hybrid_clock.cc:648] HybridClock initialized: now 1781997201122455 us; error 41 us; skew 500 ppm
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997201, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:21.125113 25230 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:13:21.125558 25230 webserver.cc:460] Webserver started at http://127.16.79.3:38521/ using document root <none> and password file <none>
I20260620 23:13:21.125783 25230 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:21.125850 25230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:21.125993 25230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:21.127014 25230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "effb69a0a0884918b246a678a5006973"
format_stamp: "Formatted at 2026-06-20 23:13:21 on dist-test-slave-pswb"
server_key: "949e5ba9e57757445a6de537fe916e90"
server_key_iv: "04b1c32659181c80b9139b4aac0f2802"
server_key_version: "encryptionkey@0"
I20260620 23:13:21.127372 25230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "effb69a0a0884918b246a678a5006973"
format_stamp: "Formatted at 2026-06-20 23:13:21 on dist-test-slave-pswb"
server_key: "949e5ba9e57757445a6de537fe916e90"
server_key_iv: "04b1c32659181c80b9139b4aac0f2802"
server_key_version: "encryptionkey@0"
I20260620 23:13:21.128683 25230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:13:21.129515 25245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.129765 25230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:21.129874 25230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "effb69a0a0884918b246a678a5006973"
format_stamp: "Formatted at 2026-06-20 23:13:21 on dist-test-slave-pswb"
server_key: "949e5ba9e57757445a6de537fe916e90"
server_key_iv: "04b1c32659181c80b9139b4aac0f2802"
server_key_version: "encryptionkey@0"
I20260620 23:13:21.129967 25230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:21.138883 25230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:21.140901 25230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:21.141052 25230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:21.141253 25230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:21.141538 25230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:21.141592 25230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.141626 25230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:21.141654 25230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.148353 25230 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:42327
I20260620 23:13:21.148404 25358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:42327 every 8 connection(s)
I20260620 23:13:21.148757 25230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997201, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:21.158447 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 25230
I20260620 23:13:21.158669 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.149854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:51489 (local address 127.16.79.62:45187)
0620 23:13:21.149946 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:21.149948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:21.150225 (+ 277us) server_negotiation.cc:407] Connection header received
0620 23:13:21.150716 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:21.150717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:21.150729 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:21.150781 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:21.151634 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.152138 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.153171 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.153365 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.153465 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.155539 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:21.155540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:21.155540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:21.155548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:21.157228 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:21.157666 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:21.157668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:21.157668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:21.157732 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:21.158007 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:21.158008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:21.158008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:21.158134 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:21.158216 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:21.158492 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 23:13:21.158570 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:13:21.158674 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:21.158864 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.149809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:51489)
0620 23:13:21.150097 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:21.150131 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:21.150611 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:21.150856 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:21.150860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:21.151374 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:21.151540 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.151544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.152232 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.152233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.153073 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.153075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.153472 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.153473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.153591 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.154146 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:21.154153 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:21.155446 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:21.157331 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:21.157337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:21.157339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:21.157570 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:21.157806 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:21.157807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:21.157808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:21.157925 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:21.158208 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:21.158214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:21.158350 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 23:13:21.158467 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:13:21.158631 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":179,"threads_started":1}
I20260620 23:13:21.158926 16700 external_mini_cluster.cc:1428] Setting key beb47183cf5d7d6e7047cf1dd4bb44ba
I20260620 23:13:21.159359 25359 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45187
I20260620 23:13:21.159461 25359 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:21.159636 25359 heartbeater.cc:511] Master 127.16.79.62:45187 requested a full tablet report, sending...
I20260620 23:13:21.160003 24899 ts_manager.cc:194] Registered new tserver with Master: effb69a0a0884918b246a678a5006973 (127.16.79.3:42327)
I20260620 23:13:21.160390 24899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:51489
I20260620 23:13:21.170835 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab.
Entry for principal oryx/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab.
WARNING: no policy specified for oryx/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab.
Entry for principal oryx/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab.
WARNING: no policy specified for oryx/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab.
Entry for principal oryx/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab.
I20260620 23:13:21.245590 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 24962
I20260620 23:13:21.250105 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 25096
I20260620 23:13:21.254225 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 25230
I20260620 23:13:21.258940 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 24868
I20260620 23:13:21.263720 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45187
--webserver_interface=127.16.79.62
--webserver_port=34527
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:45187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:21.341966 25368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:21.342154 25368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:21.342177 25368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:21.343286 25368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:21.343326 25368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:21.343341 25368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:21.343353 25368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:21.343366 25368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:21.344838 25368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:45187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=34527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:21.345057 25368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:21.345299 25368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:21.350039 25376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:21.350107 25373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:21.350061 25374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:21.350473 25368 server_base.cc:1034] running on GCE node
I20260620 23:13:21.350694 25368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:21.350911 25368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:21.352048 25368 hybrid_clock.cc:648] HybridClock initialized: now 1781997201352036 us; error 28 us; skew 500 ppm
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997201, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:21.354554 25368 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:13:21.354957 25368 webserver.cc:460] Webserver started at http://127.16.79.62:34527/ using document root <none> and password file <none>
I20260620 23:13:21.355185 25368 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:21.355263 25368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:21.356503 25368 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:21.357215 25383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.357379 25368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.357472 25368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "0479abcb05c34fe4b2e06ad404b6cb39"
format_stamp: "Formatted at 2026-06-20 23:13:20 on dist-test-slave-pswb"
server_key: "4e3a3e029ba396cb504b2cfd9ad47713"
server_key_iv: "2c5f4413ed113c9f14a8b239fb052188"
server_key_version: "encryptionkey@0"
I20260620 23:13:21.357781 25368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:21.380898 25368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:21.382926 25368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:21.383111 25368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:21.387130 25368 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:45187
I20260620 23:13:21.387169 25435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:45187 every 8 connection(s)
I20260620 23:13:21.387568 25368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:21.388935 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 25368
I20260620 23:13:21.389076 25436 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:13:21.389278 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:43313
--local_ip_for_outbound_sockets=127.16.79.1
--tserver_master_addrs=127.16.79.62:45187
--webserver_port=42379
--webserver_interface=127.16.79.1
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.79.1
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:13:21.389797 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39: Bootstrap starting.
I20260620 23:13:21.398542 25436 log.cc:824] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:21.401158 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:13:21.401477 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39: Bootstrap complete.
I20260620 23:13:21.404769 25436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45187 } }
I20260620 23:13:21.405140 25436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0479abcb05c34fe4b2e06ad404b6cb39, State: Initialized, Role: FOLLOWER
I20260620 23:13:21.405337 25436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45187 } }
I20260620 23:13:21.405437 25436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:21.405510 25436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:21.405573 25436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:13:21.406924 25436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45187 } }
I20260620 23:13:21.407065 25436 leader_election.cc:302] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0479abcb05c34fe4b2e06ad404b6cb39; no voters:
I20260620 23:13:21.407256 25436 leader_election.cc:288] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:13:21.407313 25440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:13:21.407460 25440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [term 2 LEADER]: Becoming Leader. State: Replica: 0479abcb05c34fe4b2e06ad404b6cb39, State: Running, Role: LEADER
I20260620 23:13:21.407512 25436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:21.407557 25440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45187 } }
I20260620 23:13:21.407797 25442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45187 } } }
I20260620 23:13:21.407868 25442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:21.408003 25441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0479abcb05c34fe4b2e06ad404b6cb39. Latest consensus state: current_term: 2 leader_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0479abcb05c34fe4b2e06ad404b6cb39" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45187 } } }
I20260620 23:13:21.408097 25441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:21.408031 25444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:21.408646 25444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:21.408823 25444 catalog_manager.cc:1259] Loaded cluster ID: e3ed0549ed254bd7a2061e6af349b09a
I20260620 23:13:21.408888 25444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:21.410367 25444 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:21.410576 25444 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0479abcb05c34fe4b2e06ad404b6cb39: Loaded TSK: 0
I20260620 23:13:21.410841 25444 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:13:21.477648 25438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:21.477821 25438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:21.477843 25438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:21.478961 25438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:21.479000 25438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:21.479025 25438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:21.480381 25438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.1.keytab
--principal=oryx/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:43313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=42379
--tserver_master_addrs=127.16.79.62:45187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:21.480608 25438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:21.480858 25438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:21.486483 25463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:21.486709 25438 server_base.cc:1034] running on GCE node
W20260620 23:13:21.486730 25465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:21.486495 25462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:21.487036 25438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:21.487298 25438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:21.488480 25438 hybrid_clock.cc:648] HybridClock initialized: now 1781997201488472 us; error 32 us; skew 500 ppm
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997201, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:21.491318 25438 init.cc:377] Logged in from keytab as oryx/127.16.79.1@KRBTEST.COM (short username oryx)
I20260620 23:13:21.491755 25438 webserver.cc:460] Webserver started at http://127.16.79.1:42379/ using document root <none> and password file <none>
I20260620 23:13:21.492010 25438 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:21.492090 25438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:21.493489 25438 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:13:21.494110 25472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.494302 25438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:21.494408 25438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "6490afa2333b44b7b9e7c3b4b27dfdce"
format_stamp: "Formatted at 2026-06-20 23:13:20 on dist-test-slave-pswb"
server_key: "2ebc05d041d85f2ba4e9399fbc289688"
server_key_iv: "4d7354f4ea6085b8a91ed2803296412f"
server_key_version: "encryptionkey@0"
I20260620 23:13:21.494694 25438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:21.503887 25438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:21.506258 25438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:21.506395 25438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:21.506619 25438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:21.506896 25438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:21.506950 25438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.506989 25438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:21.507017 25438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.513294 25438 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:43313
I20260620 23:13:21.513338 25585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:43313 every 8 connection(s)
I20260620 23:13:21.513666 25438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:21.514739 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 25438
I20260620 23:13:21.515060 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:38351
--local_ip_for_outbound_sockets=127.16.79.2
--tserver_master_addrs=127.16.79.62:45187
--webserver_port=36381
--webserver_interface=127.16.79.2
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.79.2
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:21.522338 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.514399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:51131)
0620 23:13:21.514659 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:21.514704 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:21.515421 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:21.515904 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:21.515908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:21.516729 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:21.516928 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.516932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.518018 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.518019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.518792 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.518794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.519367 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.519368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.519486 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.519987 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:21.520002 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:21.522135 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":86,"threads_started":1}
W20260620 23:13:21.522580 25589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:21.514668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:51131 (local address 127.16.79.62:45187)
0620 23:13:21.514829 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:21.514836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:21.514869 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:13:21.515634 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:21.515641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:21.515703 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:21.515842 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:21.517046 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.517904 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.518892 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.519277 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.519395 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.522240 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:51131: BlockingRecv error: recv got EOF from 127.16.79.1:51131 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
W20260620 23:13:21.522653 25586 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:45187 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.79.62:45187: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:21.527199 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.522851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:42039)
0620 23:13:21.522958 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:21.522968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:21.523005 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:21.523129 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:21.523129 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:21.523252 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:21.523295 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.523296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.523943 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.523944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.524438 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.524439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.525045 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.525046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.525126 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.525550 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:21.525555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:21.527126 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260620 23:13:21.527251 25589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:21.522950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:42039 (local address 127.16.79.62:45187)
0620 23:13:21.523033 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:21.523034 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:21.523042 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:13:21.523062 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:21.523062 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:21.523072 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:21.523118 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:21.523378 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.523863 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.524535 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.524942 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.525042 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.527193 (+ 2151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:42039: BlockingRecv error: recv got EOF from 127.16.79.1:42039 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:21.531523 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.527658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:35009)
0620 23:13:21.527694 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:21.527703 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:21.527739 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:21.527864 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:21.527864 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:21.527981 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:21.528004 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.528005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.528587 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.528588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.529146 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.529147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.529498 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.529499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.529549 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.529914 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:21.529919 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:21.531453 (+ 1534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 23:13:21.531572 25589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:21.527702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:35009 (local address 127.16.79.62:45187)
0620 23:13:21.527736 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:21.527738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:21.527747 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:21.527799 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:21.527800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:21.527809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:21.527851 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:21.528083 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.528510 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.529227 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.529416 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.529496 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.531515 (+ 2019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:35009: BlockingRecv error: recv got EOF from 127.16.79.1:35009 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 23:13:21.531867 25586 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:13:21.606048 25590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:21.606235 25590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:21.606256 25590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:21.607302 25590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:21.607335 25590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:21.607362 25590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:21.608940 25590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.2.keytab
--principal=oryx/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:38351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=36381
--tserver_master_addrs=127.16.79.62:45187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:21.609104 25590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:21.609265 25590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:21.613888 25595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:21.613942 25596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:21.613888 25598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:21.614228 25590 server_base.cc:1034] running on GCE node
I20260620 23:13:21.614427 25590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:21.614640 25590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:21.615796 25590 hybrid_clock.cc:648] HybridClock initialized: now 1781997201615789 us; error 35 us; skew 500 ppm
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997201, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:21.618124 25590 init.cc:377] Logged in from keytab as oryx/127.16.79.2@KRBTEST.COM (short username oryx)
I20260620 23:13:21.618556 25590 webserver.cc:460] Webserver started at http://127.16.79.2:36381/ using document root <none> and password file <none>
I20260620 23:13:21.618777 25590 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:21.618854 25590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:21.620060 25590 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:21.620575 25605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.620777 25590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:21.620862 25590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "d3263e24159c412cb2c6eccc83cc7fc2"
format_stamp: "Formatted at 2026-06-20 23:13:20 on dist-test-slave-pswb"
server_key: "2f8840a4a4e0344e5d7fc9a683bc16b2"
server_key_iv: "910730a88884049a40677c040cfd62c3"
server_key_version: "encryptionkey@0"
I20260620 23:13:21.621157 25590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:21.632783 25590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:21.634653 25590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:21.634814 25590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:21.635020 25590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:21.635288 25590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:21.635319 25590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.635377 25590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:21.635403 25590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.641091 25590 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:38351
I20260620 23:13:21.641165 25718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:38351 every 8 connection(s)
I20260620 23:13:21.641464 25590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:13:21.648344 25589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:21.642151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:41535 (local address 127.16.79.62:45187)
0620 23:13:21.642194 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:21.642195 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:21.642407 (+ 212us) server_negotiation.cc:407] Connection header received
0620 23:13:21.642883 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:21.642884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:21.642901 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:21.642951 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:21.643798 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.644291 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.645326 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.645516 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.645603 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.648269 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:41535: BlockingRecv error: recv got EOF from 127.16.79.2:41535 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:13:21.648375 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.642136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:41535)
0620 23:13:21.642301 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:21.642332 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:21.642778 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:21.643012 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:21.643015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:21.643527 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:21.643703 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.643707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.644383 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.644384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.645222 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.645224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.645625 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.645626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.645736 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.646195 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:21.646203 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:21.648200 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
W20260620 23:13:21.648631 25719 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:45187 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.79.62:45187: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
I20260620 23:13:21.651489 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 25590
I20260620 23:13:21.651813 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:42327
--local_ip_for_outbound_sockets=127.16.79.3
--tserver_master_addrs=127.16.79.62:45187
--webserver_port=38521
--webserver_interface=127.16.79.3
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.79.3
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:21.653314 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.648886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:56673)
0620 23:13:21.648935 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:21.648945 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:21.649005 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:21.649128 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:21.649128 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:21.649250 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:21.649287 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.649288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.649890 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.649891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.650377 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.650378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.650753 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.650754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.650808 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.651206 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:21.651210 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:21.653264 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 23:13:21.653432 25589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:21.648971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:56673 (local address 127.16.79.62:45187)
0620 23:13:21.649027 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:21.649030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:21.649049 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:13:21.649071 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:21.649072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:21.649080 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:21.649125 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:21.649367 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.649798 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.650457 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.650676 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.650737 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.653362 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:56673: BlockingRecv error: recv got EOF from 127.16.79.2:56673 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:21.657959 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.653737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:47155)
0620 23:13:21.653785 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:21.653796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:21.653834 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:21.653995 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:21.653996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:21.654101 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:21.654124 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.654125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.654963 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.654964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.655448 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.655450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.655967 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.655968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.656003 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.656378 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:21.656382 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:21.657921 (+ 1539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 23:13:21.658098 25589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:21.653809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:47155 (local address 127.16.79.62:45187)
0620 23:13:21.653882 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:21.653884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:21.653892 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:13:21.653915 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:21.653915 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:21.653925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:21.653993 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:21.654187 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.654886 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.655540 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.655886 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.655954 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.658021 (+ 2067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:47155: BlockingRecv error: recv got EOF from 127.16.79.2:47155 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 23:13:21.658250 25719 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:13:21.731350 25722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:21.731528 25722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:21.731550 25722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:21.732609 25722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:21.732645 25722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:21.732669 25722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:21.734064 25722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/krb5kdc/oryx_127.16.79.3.keytab
--principal=oryx/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:42327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=38521
--tserver_master_addrs=127.16.79.62:45187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:21.734244 25722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:21.734396 25722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:21.739024 25728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:21.738993 25730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:21.739213 25727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:21.739411 25722 server_base.cc:1034] running on GCE node
I20260620 23:13:21.739619 25722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:21.739842 25722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:21.740972 25722 hybrid_clock.cc:648] HybridClock initialized: now 1781997201740966 us; error 30 us; skew 500 ppm
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997201, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:21.743160 25722 init.cc:377] Logged in from keytab as oryx/127.16.79.3@KRBTEST.COM (short username oryx)
I20260620 23:13:21.743551 25722 webserver.cc:460] Webserver started at http://127.16.79.3:38521/ using document root <none> and password file <none>
I20260620 23:13:21.743778 25722 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:21.743863 25722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:21.745194 25722 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:21.745771 25737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.745970 25722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:21.746071 25722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "effb69a0a0884918b246a678a5006973"
format_stamp: "Formatted at 2026-06-20 23:13:21 on dist-test-slave-pswb"
server_key: "949e5ba9e57757445a6de537fe916e90"
server_key_iv: "04b1c32659181c80b9139b4aac0f2802"
server_key_version: "encryptionkey@0"
I20260620 23:13:21.746399 25722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:21.759554 25722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:21.761955 25722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:21.762116 25722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:21.762327 25722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:21.762610 25722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:21.762673 25722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.762717 25722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:21.762739 25722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:21.768357 25722 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:42327
I20260620 23:13:21.768432 25850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:42327 every 8 connection(s)
I20260620 23:13:21.768779 25722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:21.775662 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.769467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:45649)
0620 23:13:21.769624 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:21.769662 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:21.770099 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:21.770366 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:21.770369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:21.770885 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:21.771060 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.771064 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.771734 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.771734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.772569 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.772571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.773058 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.773059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.773158 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.773549 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:21.773556 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:21.775541 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
W20260620 23:13:21.775689 25589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:21.769491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:45649 (local address 127.16.79.62:45187)
0620 23:13:21.769550 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:21.769552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:21.769764 (+ 212us) server_negotiation.cc:407] Connection header received
0620 23:13:21.770223 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:21.770224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:21.770235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:21.770295 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:21.771161 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.771648 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.772673 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.772981 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.773046 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.775646 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:45649: BlockingRecv error: recv got EOF from 127.16.79.3:45649 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 23:13:21.775890 25851 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:45187 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.79.62:45187: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
I20260620 23:13:21.777086 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 25722
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:21.780534 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.776085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:46211)
0620 23:13:21.776174 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:21.776186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:21.776252 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:21.776474 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:21.776475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:21.776601 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:21.776645 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.776646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.777319 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.777320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.778007 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.778009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.778363 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.778364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.778426 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.778851 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:21.778857 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:21.780463 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 23:13:21.780640 25589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:21.776325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:46211 (local address 127.16.79.62:45187)
0620 23:13:21.776373 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:21.776375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:21.776381 (+ 6us) server_negotiation.cc:407] Connection header received
0620 23:13:21.776396 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:21.776397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:21.776403 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:21.776468 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:21.776739 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.777226 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.778092 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.778281 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.778362 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.780570 (+ 2208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:46211: BlockingRecv error: recv got EOF from 127.16.79.3:46211 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:21.784549 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.777472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.16.79.62:45187)
0620 23:13:21.777590 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:21.777592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:21.777654 (+ 62us) server_negotiation.cc:407] Connection header received
0620 23:13:21.777720 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:21.777721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:21.777731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:21.777791 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:21.778410 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.779041 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.779868 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.780072 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.780144 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.781474 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:21.781478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:21.781478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:21.781489 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:21.783122 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:21.783514 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:21.783517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:21.783518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:21.783614 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:21.783819 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:21.783820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:21.783820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:21.784036 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:21.784190 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:21.784300 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 23:13:21.784403 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
Jun 20 23:13:21 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:21.785168 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:21.780951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:48997)
0620 23:13:21.781000 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:21.781010 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:21.781049 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:21.781168 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:21.781168 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:21.781291 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:21.781314 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.781315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.782047 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.782048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.782666 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:21.782668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.783049 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.783050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:21.783101 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.783483 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:21.783488 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:21.785097 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 23:13:21.785235 25589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:21.781006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:48997 (local address 127.16.79.62:45187)
0620 23:13:21.781047 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:21.781048 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:21.781058 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:13:21.781101 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:21.781102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:21.781111 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:21.781164 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:21.781394 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.781946 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.782773 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:21.782964 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:21.783047 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:21.785167 (+ 2120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:48997: BlockingRecv error: recv got EOF from 127.16.79.3:48997 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 23:13:21.785393 25851 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 23:13:22 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:22 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:22.537752 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:22.532268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:58191)
0620 23:13:22.532429 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:22.532439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:22.532486 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:22.533127 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:22.533128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:22.533335 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:22.533386 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:22.533388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:22.534165 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:22.534167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:22.534780 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:22.534782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:22.535217 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:22.535219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:22.535284 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:22.535749 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:22.535754 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:22.537646 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
W20260620 23:13:22.537804 25857 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:22.532554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:58191 (local address 127.16.79.62:45187)
0620 23:13:22.532795 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:22.532797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:22.532812 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:13:22.532855 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:22.532855 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:22.532869 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:22.533121 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:22.533498 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:22.534069 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:22.534882 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:22.535101 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:22.535201 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:22.537730 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:58191: BlockingRecv error: recv got EOF from 127.16.79.1:58191 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":118,"threads_started":1}
Jun 20 23:13:22 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:22 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:22.663672 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:22.658631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:38467)
0620 23:13:22.658854 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:22.658883 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:22.658931 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:22.659249 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:22.659250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:22.659413 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:22.659461 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:22.659462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:22.660162 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:22.660162 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:22.660745 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:22.660746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:22.661246 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:22.661247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:22.661287 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:22.661793 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:22.661798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:22.663569 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":170,"threads_started":1}
W20260620 23:13:22.663707 25857 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:22.658956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:38467 (local address 127.16.79.62:45187)
0620 23:13:22.659075 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:22.659077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:22.659098 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:13:22.659129 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:22.659130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:22.659143 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:22.659228 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:22.659584 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:22.660078 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:22.660908 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:22.661178 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:22.661241 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:22.663644 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:38467: BlockingRecv error: recv got EOF from 127.16.79.2:38467 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 23:13:22 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:22 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:22.791465 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:22.785833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:36475)
0620 23:13:22.785976 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:22.785988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:22.786052 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:22.786204 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:22.786205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:22.786363 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:22.786408 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:22.786409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:22.787426 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:22.787426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:22.787946 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:22.787948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:22.788487 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:22.788488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:22.788536 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:22.789075 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:22.789079 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:22.791345 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
W20260620 23:13:22.791486 25857 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:22.785909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:36475 (local address 127.16.79.62:45187)
0620 23:13:22.785966 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:22.785968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:22.786057 (+ 89us) server_negotiation.cc:407] Connection header received
0620 23:13:22.786094 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:22.786095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:22.786114 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:22.786193 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:22.786476 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:22.787337 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:22.788026 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:22.788377 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:22.788450 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:22.791435 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:36475: BlockingRecv error: recv got EOF from 127.16.79.3:36475 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 23:13:23 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:23 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:23.543237 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:23.538345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:55929)
0620 23:13:23.538477 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:23.538486 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:23.538534 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:23.538838 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:23.538839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:23.538990 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:23.539038 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:23.539040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:23.539735 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:23.539735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:23.540249 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:23.540250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:23.540636 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:23.540637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:23.540687 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:23.541274 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:23.541278 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:23.543175 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
W20260620 23:13:23.543359 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:23.538442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:55929 (local address 127.16.79.62:45187)
0620 23:13:23.538689 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:23.538691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:23.538701 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:13:23.538726 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:23.538727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:23.538738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:23.538808 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:23.539115 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:23.539665 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:23.540336 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:23.540536 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:23.540594 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:23.543267 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:55929: BlockingRecv error: recv got EOF from 127.16.79.1:55929 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":185,"threads_started":1}
Jun 20 23:13:23 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:23 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:23.668905 25862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:23.664256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:38257)
0620 23:13:23.664454 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:23.664479 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:23.664568 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:23.664785 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:23.664786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:23.664963 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:23.665026 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:23.665027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:23.665700 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:23.665701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:23.666216 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:23.666218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:23.666589 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:23.666590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:23.666653 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:23.667077 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:23.667081 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:23.668803 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":125,"threads_started":1}
W20260620 23:13:23.668944 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:23.664319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:38257 (local address 127.16.79.62:45187)
0620 23:13:23.664443 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:23.664446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:23.664551 (+ 105us) server_negotiation.cc:407] Connection header received
0620 23:13:23.664646 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:23.664647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:23.664668 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:23.664771 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:23.665110 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:23.665599 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:23.666313 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:23.666510 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:23.666579 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:23.668881 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:38257: BlockingRecv error: recv got EOF from 127.16.79.2:38257 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 23:13:23 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:23 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:23.796875 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:23.791999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:43121)
0620 23:13:23.792145 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:23.792157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:23.792221 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:23.792453 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:23.792454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:23.792626 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:23.792670 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:23.792672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:23.793379 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:23.793380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:23.793916 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:23.793917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:23.794285 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:23.794286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:23.794342 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:23.794818 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:23.794824 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:23.796770 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":56,"threads_started":1}
W20260620 23:13:23.796917 25861 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:23.792225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:43121 (local address 127.16.79.62:45187)
0620 23:13:23.792308 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:23.792310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:23.792321 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:13:23.792355 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:23.792355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:23.792367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:23.792431 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:23.792791 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:23.793288 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:23.794005 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:23.794204 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:23.794276 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:23.796854 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:43121: BlockingRecv error: recv got EOF from 127.16.79.3:43121 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 23:13:24 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:24 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:24.548871 25864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:24.543773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:48107)
0620 23:13:24.543905 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:24.543918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:24.543983 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:24.544143 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:24.544144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:24.544303 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:24.544352 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:24.544354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:24.545118 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:24.545120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:24.545759 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:24.545762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:24.546217 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:24.546218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:24.546285 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:24.546840 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:24.546847 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:24.548776 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
W20260620 23:13:24.548902 25865 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:24.543817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:48107 (local address 127.16.79.62:45187)
0620 23:13:24.543958 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:24.543960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:24.543969 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:24.544045 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:24.544046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:24.544055 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:24.544119 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:24.544420 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:24.544947 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:24.545904 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:24.546101 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:24.546170 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:24.548822 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:48107: BlockingRecv error: recv got EOF from 127.16.79.1:48107 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
Jun 20 23:13:24 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:24 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:24.675275 25866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:24.669547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:39577)
0620 23:13:24.669826 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:24.669843 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:24.669951 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:24.670129 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:24.670130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:24.670330 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:24.670387 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:24.670389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:24.671062 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:24.671063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:24.671588 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:24.671589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:24.671989 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:24.671991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:24.672061 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:24.672562 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:24.672568 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:24.675158 (+ 2590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":137,"threads_started":1}
W20260620 23:13:24.675297 25865 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:24.669711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:39577 (local address 127.16.79.62:45187)
0620 23:13:24.669831 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:24.669834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:24.669926 (+ 92us) server_negotiation.cc:407] Connection header received
0620 23:13:24.670025 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:24.670026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:24.670040 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:24.670120 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:24.670474 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:24.670976 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:24.671679 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:24.671895 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:24.671983 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:24.675248 (+ 3265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:39577: BlockingRecv error: recv got EOF from 127.16.79.2:39577 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 23:13:24 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:24 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:24.802443 25867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:24.797372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:45755)
0620 23:13:24.797517 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:24.797529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:24.797608 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:24.797789 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:24.797790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:24.797982 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:24.798033 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:24.798035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:24.798852 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:24.798853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:24.799435 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:24.799437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:24.799808 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:24.799809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:24.799855 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:24.800345 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:24.800350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:24.802374 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
W20260620 23:13:24.802551 25865 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:24.797456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:45755 (local address 127.16.79.62:45187)
0620 23:13:24.797528 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:24.797530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:24.797586 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:13:24.797688 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:24.797689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:24.797702 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:24.797767 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:24.798234 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:24.798776 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:24.799535 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:24.799727 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:24.799790 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:24.802471 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:45755: BlockingRecv error: recv got EOF from 127.16.79.3:45755 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 23:13:25 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:25 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:25.555095 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:25.549553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:54357)
0620 23:13:25.549737 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:25.549750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:25.549814 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:25.550090 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:25.550090 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:25.550349 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:25.550420 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:25.550422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:25.551396 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:25.551398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:25.552010 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:25.552012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:25.552428 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:25.552429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:25.552488 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:25.553071 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:25.553076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:25.555040 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
W20260620 23:13:25.555217 25869 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:25.549695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:54357 (local address 127.16.79.62:45187)
0620 23:13:25.549898 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:25.549900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:25.549921 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:13:25.549955 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:25.549956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:25.549971 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:25.550068 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:25.550526 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:25.551299 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:25.552106 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:25.552343 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:25.552430 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:25.555133 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:54357: BlockingRecv error: recv got EOF from 127.16.79.1:54357 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
Jun 20 23:13:25 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:25 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:25.683908 25870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:25.676031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:38305)
0620 23:13:25.676171 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:25.676182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:25.676249 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:25.676390 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:25.676391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:25.676546 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:25.676592 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:25.676593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:25.677348 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:25.677349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:25.678024 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:25.678026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:25.678397 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:25.678398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:25.678437 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:25.681704 (+ 3267us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:25.681717 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:25.683853 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":97,"threads_started":1}
W20260620 23:13:25.684073 25869 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:25.676091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:38305 (local address 127.16.79.62:45187)
0620 23:13:25.676167 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:25.676170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:25.676249 (+ 79us) server_negotiation.cc:407] Connection header received
0620 23:13:25.676310 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:25.676310 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:25.676322 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:25.676370 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:25.676704 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:25.677244 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:25.678120 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:25.678330 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:25.678386 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:25.683971 (+ 5585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:38305: BlockingRecv error: recv got EOF from 127.16.79.2:38305 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 23:13:25 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:25 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:25.808687 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:25.802953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:34083)
0620 23:13:25.803173 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:25.803191 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:25.803296 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:25.803697 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:25.803700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:25.803939 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:25.804023 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:25.804026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:25.804989 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:25.804990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:25.805528 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:25.805530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:25.806005 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:25.806007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:25.806079 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:25.806633 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:25.806638 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:25.808583 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":56,"threads_started":1}
W20260620 23:13:25.808867 25869 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:25.803341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:34083 (local address 127.16.79.62:45187)
0620 23:13:25.803422 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:25.803424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:25.803439 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:13:25.803479 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:25.803479 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:25.803496 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:25.803579 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:25.804150 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:25.804903 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:25.805665 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:25.805877 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:25.805946 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:25.808778 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:34083: BlockingRecv error: recv got EOF from 127.16.79.3:34083 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 23:13:26 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:26 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:26.560549 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:26.555591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:57143)
0620 23:13:26.555713 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:26.555731 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:26.555777 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:26.556221 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:26.556222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:26.556376 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:26.556428 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:26.556430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:26.557313 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:26.557314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:26.557840 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:26.557841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:26.558218 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:26.558220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:26.558269 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:26.558736 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:26.558740 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:26.560484 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
W20260620 23:13:26.560921 25873 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:26.555836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:57143 (local address 127.16.79.62:45187)
0620 23:13:26.556023 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:26.556026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:26.556037 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:13:26.556070 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:26.556071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:26.556085 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:26.556188 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:26.556602 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:26.557220 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:26.557939 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:26.558129 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:26.558197 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:26.560814 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:57143: BlockingRecv error: recv got EOF from 127.16.79.1:57143 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":117,"threads_started":1}
Jun 20 23:13:26 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:26 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:26.689042 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:26.684454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:35969)
0620 23:13:26.684603 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:26.684615 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:26.684674 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:26.684891 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:26.684892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:26.685075 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:26.685120 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:26.685122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:26.685778 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:26.685779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:26.686302 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:26.686303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:26.686680 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:26.686681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:26.686740 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:26.687216 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:26.687220 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:26.688951 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
W20260620 23:13:26.689106 25873 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:26.684589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:35969 (local address 127.16.79.62:45187)
0620 23:13:26.684734 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:26.684736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:26.684745 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:26.684774 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:26.684774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:26.684787 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:26.684882 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:26.685201 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:26.685696 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:26.686397 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:26.686586 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:26.686679 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:26.689024 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:35969: BlockingRecv error: recv got EOF from 127.16.79.2:35969 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 23:13:26 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:26 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:26.814777 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:26.809135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:33937)
0620 23:13:26.809351 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:26.809368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:26.809498 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:26.809682 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:26.809683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:26.809972 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:26.810042 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:26.810044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:26.810840 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:26.810841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:26.811560 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:26.811572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:26.812026 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:26.812027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:26.812082 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:26.812596 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:26.812601 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:26.814717 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
W20260620 23:13:26.814852 25873 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:26.809323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:33937 (local address 127.16.79.62:45187)
0620 23:13:26.809466 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:26.809468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:26.809482 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:13:26.809571 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:26.809571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:26.809591 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:26.809677 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:26.810143 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:26.810747 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:26.811697 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:26.811933 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:26.812026 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:26.814800 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:33937: BlockingRecv error: recv got EOF from 127.16.79.3:33937 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Jun 20 23:13:27 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:27 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:27.566326 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:27.561050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:58593)
0620 23:13:27.561205 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:27.561217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:27.561283 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:27.561461 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:27.561461 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:27.561638 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:27.561682 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:27.561684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:27.562409 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:27.562410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:27.562935 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:27.562936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:27.563330 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:27.563331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:27.563393 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:27.563988 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:27.563992 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:27.566242 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
W20260620 23:13:27.566468 25877 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:27.561089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:58593 (local address 127.16.79.62:45187)
0620 23:13:27.561216 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:27.561218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:27.561265 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:13:27.561364 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:27.561365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:27.561377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:27.561455 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:27.561765 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:27.562276 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:27.563032 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:27.563239 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:27.563328 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:27.566340 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:58593: BlockingRecv error: recv got EOF from 127.16.79.1:58593 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":43,"threads_started":1}
Jun 20 23:13:27 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:27 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:27.695168 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:27.689528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:53655)
0620 23:13:27.689678 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:27.689797 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:27.689931 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:27.690199 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:27.690200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:27.690359 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:27.690437 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:27.690439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:27.691398 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:27.691399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:27.691965 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:27.691967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:27.692321 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:27.692322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:27.692383 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:27.692947 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:27.692952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:27.695102 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":113,"threads_started":1}
W20260620 23:13:27.695334 25877 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:27.689588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:53655 (local address 127.16.79.62:45187)
0620 23:13:27.689657 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:27.689658 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:27.689949 (+ 291us) server_negotiation.cc:407] Connection header received
0620 23:13:27.689993 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:27.689994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:27.690015 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:27.690090 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:27.690561 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:27.691307 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:27.692044 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:27.692246 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:27.692307 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:27.695250 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:53655: BlockingRecv error: recv got EOF from 127.16.79.2:53655 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 23:13:27 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:27 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:27.819823 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:27.815320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:56185)
0620 23:13:27.815461 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:27.815472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:27.815538 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:27.815726 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:27.815727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:27.815887 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:27.815934 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:27.815936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:27.816587 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:27.816588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:27.817138 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:27.817139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:27.817511 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:27.817512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:27.817549 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:27.817998 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:27.818003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:27.819757 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
W20260620 23:13:27.819924 25877 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:27.815391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:56185 (local address 127.16.79.62:45187)
0620 23:13:27.815468 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:27.815470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:27.815584 (+ 114us) server_negotiation.cc:407] Connection header received
0620 23:13:27.815610 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:27.815611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:27.815624 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:27.815672 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:27.816017 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:27.816503 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:27.817227 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:27.817433 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:27.817504 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:27.819853 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:56185: BlockingRecv error: recv got EOF from 127.16.79.3:56185 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 23:13:28 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:28 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:28.571432 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:28.566878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:35665)
0620 23:13:28.567031 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:28.567048 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:28.567103 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:28.567241 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:28.567242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:28.567401 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:28.567451 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:28.567452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:28.568075 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:28.568075 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:28.568601 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:28.568602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:28.568984 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:28.568985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:28.569029 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:28.569448 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:28.569452 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:28.571371 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
W20260620 23:13:28.571537 25881 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:28.566926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:35665 (local address 127.16.79.62:45187)
0620 23:13:28.567059 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:28.567061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:28.567088 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:13:28.567155 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:28.567155 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:28.567165 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:28.567224 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:28.567511 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:28.568011 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:28.568673 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:28.568910 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:28.568966 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:28.571461 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:35665: BlockingRecv error: recv got EOF from 127.16.79.1:35665 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
Jun 20 23:13:28 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:28 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:28.700918 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:28.695648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:45731)
0620 23:13:28.695757 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:28.695774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:28.695834 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:28.696118 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:28.696119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:28.696273 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:28.696322 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:28.696323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:28.697107 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:28.697108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:28.697692 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:28.697694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:28.698350 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:28.698351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:28.698436 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:28.698958 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:28.698963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:28.700860 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
W20260620 23:13:28.701197 25881 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:28.695650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:45731 (local address 127.16.79.62:45187)
0620 23:13:28.695702 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:28.695704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:28.695916 (+ 212us) server_negotiation.cc:407] Connection header received
0620 23:13:28.695977 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:28.695979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:28.696002 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:28.696104 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:28.696426 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:28.697021 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:28.697816 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:28.698230 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:28.698306 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:28.701105 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:45731: BlockingRecv error: recv got EOF from 127.16.79.2:45731 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 23:13:28 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:28 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:28.824962 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:28.820438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:49753)
0620 23:13:28.820576 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:28.820593 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:28.820644 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:28.820831 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:28.820831 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:28.820986 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:28.821034 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:28.821035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:28.821712 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:28.821713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:28.822224 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:28.822226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:28.822561 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:28.822562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:28.822608 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:28.823133 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:28.823137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:28.824906 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
W20260620 23:13:28.825115 25881 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:28.820539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:49753 (local address 127.16.79.62:45187)
0620 23:13:28.820629 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:28.820635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:28.820646 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:13:28.820744 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:28.820745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:28.820759 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:28.820814 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:28.821141 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:28.821631 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:28.822300 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:28.822472 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:28.822528 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:28.825015 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:49753: BlockingRecv error: recv got EOF from 127.16.79.3:49753 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 23:13:29 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:29 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:29.577105 25884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:29.572213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:35569)
0620 23:13:29.572386 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:29.572398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:29.572449 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:29.572648 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:29.572649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:29.572867 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:29.572917 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:29.572919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:29.573607 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:29.573608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:29.574120 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:29.574121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:29.574511 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:29.574512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:29.574555 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:29.575048 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:29.575052 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:29.577049 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":54,"threads_started":1}
W20260620 23:13:29.577208 25885 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:29.572399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:35569 (local address 127.16.79.62:45187)
0620 23:13:29.572521 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:29.572523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:29.572538 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:13:29.572558 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:29.572559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:29.572568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:29.572626 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:29.573013 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:29.573518 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:29.574207 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:29.574413 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:29.574470 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:29.577133 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:35569: BlockingRecv error: recv got EOF from 127.16.79.1:35569 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
Jun 20 23:13:29 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:29 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:29.707330 25886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:29.701454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:57579)
0620 23:13:29.701598 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:29.701615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:29.701663 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:29.701821 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:29.701822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:29.701968 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:29.702015 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:29.702016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:29.703011 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:29.703012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:29.703837 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:29.703841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:29.704349 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:29.704350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:29.704408 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:29.704947 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:29.704952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:29.707239 (+ 2287us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
W20260620 23:13:29.707373 25885 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:29.701647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:57579 (local address 127.16.79.62:45187)
0620 23:13:29.701711 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:29.701713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:29.701721 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:13:29.701745 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:29.701745 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:29.701755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:29.701807 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:29.702074 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:29.702911 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:29.703961 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:29.704268 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:29.704337 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:29.707303 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:57579: BlockingRecv error: recv got EOF from 127.16.79.2:57579 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 23:13:29 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:29 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:29.830582 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:29.825411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:43871)
0620 23:13:29.825542 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:29.825552 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:29.825624 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:29.825746 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:29.825747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:29.825898 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:29.825942 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:29.825943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:29.826677 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:29.826678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:29.827563 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:29.827570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:29.828090 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:29.828091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:29.828171 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:29.828672 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:29.828677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:29.830480 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
W20260620 23:13:29.830627 25885 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:29.825470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:43871 (local address 127.16.79.62:45187)
0620 23:13:29.825551 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:29.825552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:29.825637 (+ 85us) server_negotiation.cc:407] Connection header received
0620 23:13:29.825665 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:29.825665 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:29.825679 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:29.825734 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:29.826035 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:29.826594 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:29.827695 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:29.827997 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:29.828088 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:29.830558 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:43871: BlockingRecv error: recv got EOF from 127.16.79.3:43871 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 23:13:30 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:30 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:30.582499 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:30.577556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:42651)
0620 23:13:30.577684 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:30.577695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:30.577756 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:30.578043 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:30.578045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:30.578282 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:30.578352 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:30.578354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:30.579096 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:30.579097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:30.579674 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:30.579675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:30.580025 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:30.580026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:30.580063 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:30.580523 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:30.580527 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:30.582443 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
W20260620 23:13:30.582613 25889 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:30.577637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:42651 (local address 127.16.79.62:45187)
0620 23:13:30.577783 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:30.577785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:30.577792 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:30.577846 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:30.577847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:30.577857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:30.577920 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:30.578484 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:30.578991 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:30.579762 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:30.579957 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:30.580013 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:30.582563 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:42651: BlockingRecv error: recv got EOF from 127.16.79.1:42651 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
Jun 20 23:13:30 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:30 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:30.715901 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:30.707891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:39315)
0620 23:13:30.708040 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:30.708051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:30.708113 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:30.708275 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:30.708276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:30.708431 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:30.708479 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:30.708481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:30.709190 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:30.709191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:30.709701 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:30.709702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:30.710379 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:30.710379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:30.710426 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:30.713982 (+ 3556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:30.713987 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:30.715837 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
W20260620 23:13:30.716049 25889 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:30.707954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:39315 (local address 127.16.79.62:45187)
0620 23:13:30.708000 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:30.708002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:30.708114 (+ 112us) server_negotiation.cc:407] Connection header received
0620 23:13:30.708157 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:30.708157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:30.708168 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:30.708267 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:30.708538 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:30.709122 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:30.709774 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:30.710238 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:30.710368 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:30.715961 (+ 5593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:39315: BlockingRecv error: recv got EOF from 127.16.79.2:39315 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 23:13:30 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:30 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:30.835705 25891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:30.831059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:43653)
0620 23:13:30.831227 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:30.831247 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:30.831309 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:30.831443 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:30.831444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:30.831600 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:30.831645 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:30.831646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:30.832429 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:30.832429 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:30.832949 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:30.832951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:30.833301 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:30.833301 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:30.833355 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:30.833806 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:30.833811 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:30.835649 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":53,"threads_started":1}
W20260620 23:13:30.835861 25889 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:30.831176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:43653 (local address 127.16.79.62:45187)
0620 23:13:30.831256 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:30.831258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:30.831295 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:13:30.831355 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:30.831355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:30.831368 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:30.831420 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:30.831708 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:30.832203 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:30.833047 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:30.833233 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:30.833293 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:30.835759 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:43653: BlockingRecv error: recv got EOF from 127.16.79.3:43653 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 23:13:31 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:31 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:31.588279 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:31.583104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:36559)
0620 23:13:31.583290 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:31.583303 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:31.583365 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:31.583560 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:31.583561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:31.583788 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:31.583849 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:31.583852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:31.584605 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:31.584607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:31.585429 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:31.585432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:31.585845 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:31.585846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:31.585898 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:31.586375 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:31.586379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:31.588219 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
W20260620 23:13:31.588539 25893 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:31.583252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:36559 (local address 127.16.79.62:45187)
0620 23:13:31.583374 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:31.583376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:31.583387 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:13:31.583460 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:31.583461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:31.583474 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:31.583541 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:31.583939 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:31.584440 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:31.585531 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:31.585752 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:31.585813 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:31.588344 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:36559: BlockingRecv error: recv got EOF from 127.16.79.1:36559 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
Jun 20 23:13:31 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:31 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:31.720873 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:31.716346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:57587)
0620 23:13:31.716528 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:31.716542 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:31.716636 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:31.716847 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:31.716848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:31.717018 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:31.717063 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:31.717064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:31.717723 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:31.717724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:31.718241 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:31.718242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:31.718619 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:31.718620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:31.718685 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:31.719100 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:31.719104 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:31.720816 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
W20260620 23:13:31.721041 25893 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:31.716439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:57587 (local address 127.16.79.62:45187)
0620 23:13:31.716533 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:31.716535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:31.716617 (+ 82us) server_negotiation.cc:407] Connection header received
0620 23:13:31.716752 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:31.716752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:31.716765 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:31.716842 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:31.717143 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:31.717640 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:31.718339 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:31.718535 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:31.718619 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:31.720973 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:57587: BlockingRecv error: recv got EOF from 127.16.79.2:57587 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 23:13:31 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:31 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:31.841121 25895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:31.836184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:49003)
0620 23:13:31.836338 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:31.836352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:31.836418 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:31.836576 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:31.836577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:31.836817 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:31.836860 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:31.836861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:31.837504 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:31.837505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:31.838051 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:31.838052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:31.838410 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:31.838410 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:31.838463 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:31.838938 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:31.838942 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:31.841030 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
W20260620 23:13:31.841167 25893 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:31.836306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:49003 (local address 127.16.79.62:45187)
0620 23:13:31.836359 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:31.836361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:31.836400 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:13:31.836478 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:31.836479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:31.836494 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:31.836568 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:31.836935 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:31.837423 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:31.838135 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:31.838335 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:31.838413 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:31.841102 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:49003: BlockingRecv error: recv got EOF from 127.16.79.3:49003 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 23:13:32 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:32 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:32.593673 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:32.589039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:49157)
0620 23:13:32.589167 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:32.589179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:32.589227 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:32.589412 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:32.589413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:32.589569 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:32.589611 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:32.589612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:32.590252 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:32.590253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:32.590817 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:32.590818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:32.591162 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:32.591163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:32.591203 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:32.591678 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:32.591683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:32.593604 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
W20260620 23:13:32.593741 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:32.589097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:49157 (local address 127.16.79.62:45187)
0620 23:13:32.589216 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:32.589218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:32.589225 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:32.589295 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:32.589295 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:32.589306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:32.589372 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:32.589680 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:32.590180 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:32.590905 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:32.591092 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:32.591151 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:32.593689 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:49157: BlockingRecv error: recv got EOF from 127.16.79.1:49157 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":48,"threads_started":1}
Jun 20 23:13:32 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:32 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:32.726096 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:32.721294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:42313)
0620 23:13:32.721467 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:32.721480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:32.721580 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:32.721742 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:32.721742 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:32.721918 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:32.721962 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:32.721964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:32.722649 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:32.722650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:32.723240 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:32.723242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:32.723660 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:32.723661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:32.723722 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:32.724250 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:32.724254 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:32.725993 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
W20260620 23:13:32.726131 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:32.721404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:42313 (local address 127.16.79.62:45187)
0620 23:13:32.721475 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:32.721477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:32.721559 (+ 82us) server_negotiation.cc:407] Connection header received
0620 23:13:32.721668 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:32.721669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:32.721681 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:32.721723 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:32.722038 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:32.722562 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:32.723327 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:32.723577 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:32.723659 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:32.726067 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:42313: BlockingRecv error: recv got EOF from 127.16.79.2:42313 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 23:13:32 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:32 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:32.846377 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:32.841618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:47295)
0620 23:13:32.841742 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:32.841753 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:32.841804 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:32.841913 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:32.841914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:32.842069 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:32.842125 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:32.842126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:32.842783 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:32.842784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:32.843320 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:32.843321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:32.843716 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:32.843717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:32.843761 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:32.844504 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:32.844509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:32.846302 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
W20260620 23:13:32.846529 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:32.841689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:47295 (local address 127.16.79.62:45187)
0620 23:13:32.841779 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:32.841781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:32.841790 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:32.841856 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:32.841857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:32.841868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:32.841902 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:32.842200 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:32.842691 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:32.843441 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:32.843621 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:32.843681 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:32.846427 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:47295: BlockingRecv error: recv got EOF from 127.16.79.3:47295 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 23:13:33 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:33 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:33.599123 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:33.594217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:37657)
0620 23:13:33.594368 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:33.594378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:33.594425 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:33.594552 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:33.594553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:33.594708 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:33.594753 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:33.594754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:33.595498 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:33.595499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:33.596037 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:33.596039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:33.596414 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:33.596415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:33.596498 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:33.597108 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:33.597113 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:33.599021 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
W20260620 23:13:33.599153 25901 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:33.594272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:37657 (local address 127.16.79.62:45187)
0620 23:13:33.594425 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:33.594427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:33.594435 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:13:33.594475 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:33.594475 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:33.594485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:33.594537 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:33.594909 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:33.595412 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:33.596112 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:33.596293 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:33.596377 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:33.599083 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:37657: BlockingRecv error: recv got EOF from 127.16.79.1:37657 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
Jun 20 23:13:33 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:33 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:33.731355 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:33.726541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:33109)
0620 23:13:33.726659 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:33.726677 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:33.726724 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:33.726852 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:33.726852 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:33.727020 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:33.727066 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:33.727067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:33.727749 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:33.727751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:33.728458 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:33.728463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:33.728787 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:33.728788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:33.728837 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:33.729353 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:33.729358 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:33.731259 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
W20260620 23:13:33.731384 25901 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:33.726636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:33109 (local address 127.16.79.62:45187)
0620 23:13:33.726712 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:33.726713 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:33.726721 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:13:33.726782 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:33.726783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:33.726794 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:33.726837 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:33.727126 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:33.727620 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:33.728539 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:33.728734 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:33.728784 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:33.731328 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:33109: BlockingRecv error: recv got EOF from 127.16.79.2:33109 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 23:13:33 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:33 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:33.851668 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:33.846877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:35271)
0620 23:13:33.847056 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:33.847068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:33.847155 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:33.847302 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:33.847303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:33.847485 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:33.847527 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:33.847528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:33.848196 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:33.848197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:33.848832 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:33.848834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:33.849191 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:33.849192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:33.849246 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:33.849728 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:33.849734 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:33.851591 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":97,"threads_started":1}
W20260620 23:13:33.851768 25901 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:33.846965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:35271 (local address 127.16.79.62:45187)
0620 23:13:33.847038 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:33.847040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:33.847136 (+ 96us) server_negotiation.cc:407] Connection header received
0620 23:13:33.847208 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:33.847209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:33.847222 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:33.847297 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:33.847605 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:33.848115 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:33.848917 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:33.849107 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:33.849189 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:33.851692 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:35271: BlockingRecv error: recv got EOF from 127.16.79.3:35271 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 23:13:34 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:34 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:34.605178 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:34.599693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:37637)
0620 23:13:34.599864 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:34.599879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:34.599945 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:34.600184 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:34.600186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:34.600431 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:34.600499 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:34.600502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:34.601352 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:34.601354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:34.602148 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:34.602150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:34.602733 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:34.602734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:34.602779 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:34.603281 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:34.603285 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:34.605106 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
W20260620 23:13:34.605335 25905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:34.599761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:37637 (local address 127.16.79.62:45187)
0620 23:13:34.599935 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:34.599937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:34.599946 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:13:34.600005 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:34.600005 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:34.600016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:34.600066 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:34.600643 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:34.601248 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:34.602282 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:34.602616 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:34.602687 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:34.605245 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:37637: BlockingRecv error: recv got EOF from 127.16.79.1:37637 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":111,"threads_started":1}
Jun 20 23:13:34 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:34 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:34.736063 25906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:34.731786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:56833)
0620 23:13:34.731925 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:34.731935 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:34.731993 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:34.732110 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:34.732110 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:34.732261 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:34.732305 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:34.732306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:34.733028 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:34.733029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:34.733559 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:34.733561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:34.733894 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:34.733895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:34.733932 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:34.734342 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:34.734346 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:34.736008 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":50,"threads_started":1}
W20260620 23:13:34.736205 25905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:34.731898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:56833 (local address 127.16.79.62:45187)
0620 23:13:34.731943 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:34.731944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:34.731979 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:13:34.732035 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:34.732036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:34.732045 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:34.732093 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:34.732388 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:34.732960 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:34.733632 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:34.733828 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:34.733885 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:34.736123 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:56833: BlockingRecv error: recv got EOF from 127.16.79.2:56833 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Jun 20 23:13:34 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:34 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:34.857067 25907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:34.852156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:34889)
0620 23:13:34.852293 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:34.852305 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:34.852374 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:34.852499 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:34.852500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:34.852658 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:34.852762 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:34.852767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:34.853427 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:34.853428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:34.853966 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:34.853968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:34.854266 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:34.854267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:34.854303 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:34.854775 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:34.854781 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:34.857000 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
W20260620 23:13:34.857180 25905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:34.852230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:34889 (local address 127.16.79.62:45187)
0620 23:13:34.852276 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:34.852278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:34.852379 (+ 101us) server_negotiation.cc:407] Connection header received
0620 23:13:34.852415 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:34.852416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:34.852428 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:34.852479 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:34.852865 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:34.853353 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:34.854039 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:34.854208 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:34.854256 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:34.857090 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:34889: BlockingRecv error: recv got EOF from 127.16.79.3:34889 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 23:13:35 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:35 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:35.610926 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:35.605604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:36899)
0620 23:13:35.605773 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:35.605785 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:35.605863 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:35.606000 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:35.606000 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:35.606175 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:35.606225 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:35.606227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:35.606872 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:35.606873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:35.607472 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:35.607474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:35.607902 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:35.607904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:35.607958 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:35.608654 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:35.608661 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:35.610828 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
W20260620 23:13:35.610983 25909 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:35.605690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:36899 (local address 127.16.79.62:45187)
0620 23:13:35.605787 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:35.605789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:35.605868 (+ 79us) server_negotiation.cc:407] Connection header received
0620 23:13:35.605912 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:35.605912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:35.605926 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:35.605982 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:35.606285 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:35.606800 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:35.607581 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:35.607830 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:35.607905 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:35.610912 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:36899: BlockingRecv error: recv got EOF from 127.16.79.1:36899 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":47,"threads_started":1}
Jun 20 23:13:35 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:35 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:35.744139 25910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:35.736496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:41065)
0620 23:13:35.736677 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:35.736724 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:35.736784 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:35.736953 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:35.736955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:35.737175 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:35.737234 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:35.737236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:35.737943 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:35.737944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:35.738493 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:35.738495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:35.738843 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:35.738844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:35.738881 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:35.742416 (+ 3535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:35.742424 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:35.744082 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
W20260620 23:13:35.744231 25909 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:35.736595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:41065 (local address 127.16.79.62:45187)
0620 23:13:35.736676 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:35.736678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:35.736793 (+ 115us) server_negotiation.cc:407] Connection header received
0620 23:13:35.736847 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:35.736848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:35.736862 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:35.736939 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:35.737328 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:35.737863 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:35.738581 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:35.738775 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:35.738836 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:35.744167 (+ 5331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:41065: BlockingRecv error: recv got EOF from 127.16.79.2:41065 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 23:13:35 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:35 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.3@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:35.861820 25911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:35.857486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.3:48237)
0620 23:13:35.857671 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:35.857681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:35.857735 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:35.857862 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:35.857862 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:35.858024 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:35.858068 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:35.858069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:35.858684 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:35.858684 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:35.859193 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:35.859194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:35.859492 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:35.859493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:35.859527 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:35.859990 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:35.859994 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:35.861721 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
W20260620 23:13:35.861857 25909 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:35.857569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:48237 (local address 127.16.79.62:45187)
0620 23:13:35.857638 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:35.857640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:35.857736 (+ 96us) server_negotiation.cc:407] Connection header received
0620 23:13:35.857775 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:35.857776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:35.857789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:35.857841 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:35.858125 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:35.858618 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:35.859263 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:35.859434 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:35.859482 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:35.861809 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.3:48237: BlockingRecv error: recv got EOF from 127.16.79.3:48237 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 23:13:36 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:36 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.1@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:36.616168 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:36.611439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.1:46841)
0620 23:13:36.611631 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:36.611654 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:36.611747 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:36.611962 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:36.611963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:36.612120 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:36.612178 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:36.612180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:36.612996 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:36.612997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:36.613504 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:36.613505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:36.613862 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:36.613863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:36.613908 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:36.614383 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:36.614388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:36.616102 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":57,"threads_started":1}
W20260620 23:13:36.616314 25913 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:36.611648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:46841 (local address 127.16.79.62:45187)
0620 23:13:36.611771 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:36.611774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:36.611800 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:13:36.611822 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:36.611822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:36.611833 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:36.611919 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:36.612304 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:36.612920 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:36.613594 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:36.613792 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:36.613850 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:36.616197 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.1:46841: BlockingRecv error: recv got EOF from 127.16.79.1:46841 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
Jun 20 23:13:36 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:13:36 dist-test-slave-pswb krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.79.2@KRBTEST.COM for oryx/127.16.79.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:13:36.749706 25914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:36.744671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45187 (local address 127.16.79.2:36781)
0620 23:13:36.744831 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:36.744849 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:36.744925 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:36.745062 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:36.745063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:36.745232 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:36.745278 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:36.745279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:36.745950 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:36.745951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:36.746467 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:36.746468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:36.746858 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:36.746859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:36.746925 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:36.747425 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:36.747429 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:36.749651 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.79.62:45187: Server oryx/127.16.79.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
W20260620 23:13:36.749841 25913 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:36.744827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:36781 (local address 127.16.79.62:45187)
0620 23:13:36.744923 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:36.744925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:36.744935 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:13:36.744986 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:36.744988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:36.744999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:36.745044 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:36.745359 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:36.745856 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:36.746552 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:36.746770 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:36.746855 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:36.749752 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.79.2:36781: BlockingRecv error: recv got EOF from 127.16.79.2:36781 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:13:36.781226 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 25438
I20260620 23:13:36.785892 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 25590
I20260620 23:13:36.790127 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 25722
I20260620 23:13:36.794260 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 25368
2026-06-20T23:13:36Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17410 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T23:13:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:36Z Disabled control of system clock
I20260620 23:13:36.822153 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45393
--webserver_interface=127.16.79.62
--webserver_port=0
--builtin_ntp_servers=127.16.79.20:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:45393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:13:36.904870 25918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:36.905081 25918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:36.905099 25918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:36.906194 25918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:36.906232 25918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:36.906245 25918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:36.906257 25918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:36.906270 25918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:36.907500 25918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:45393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:36.907685 25918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:36.907907 25918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:13:36.913776 25918 server_base.cc:1034] running on GCE node
W20260620 23:13:36.913748 25923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:36.913748 25924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:36.913795 25926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:36.914186 25918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:36.914486 25918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:36.915670 25918 hybrid_clock.cc:648] HybridClock initialized: now 1781997216915644 us; error 62 us; skew 500 ppm
I20260620 23:13:36.917045 25918 webserver.cc:460] Webserver started at http://127.16.79.62:33047/ using document root <none> and password file <none>
I20260620 23:13:36.917284 25918 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:36.917356 25918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:36.917496 25918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:36.918623 25918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "03e54f1ef93640c99690a339b477f082"
format_stamp: "Formatted at 2026-06-20 23:13:36 on dist-test-slave-pswb"
server_key: "319ec8d7ddfb52d42d4a56333a54e1f0"
server_key_iv: "34de875ed82a814526e2590f0f3671da"
server_key_version: "encryptionkey@0"
I20260620 23:13:36.918993 25918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "03e54f1ef93640c99690a339b477f082"
format_stamp: "Formatted at 2026-06-20 23:13:36 on dist-test-slave-pswb"
server_key: "319ec8d7ddfb52d42d4a56333a54e1f0"
server_key_iv: "34de875ed82a814526e2590f0f3671da"
server_key_version: "encryptionkey@0"
I20260620 23:13:36.920305 25918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:13:36.921140 25932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:36.921324 25918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:13:36.921420 25918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "03e54f1ef93640c99690a339b477f082"
format_stamp: "Formatted at 2026-06-20 23:13:36 on dist-test-slave-pswb"
server_key: "319ec8d7ddfb52d42d4a56333a54e1f0"
server_key_iv: "34de875ed82a814526e2590f0f3671da"
server_key_version: "encryptionkey@0"
I20260620 23:13:36.921499 25918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:36.938642 25918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:36.938949 25918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:36.939107 25918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:36.943467 25918 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:45393
I20260620 23:13:36.943662 25984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:45393 every 8 connection(s)
I20260620 23:13:36.943885 25918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:36.944546 25985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:36.946400 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 25918
I20260620 23:13:36.946478 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:36.946859 16700 external_mini_cluster.cc:1428] Setting key 1bb4e2fdf7d178fe07607c19107ecbda
I20260620 23:13:36.948685 25985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082: Bootstrap starting.
I20260620 23:13:36.949489 25985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:36.949944 25985 log.cc:824] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:36.950965 25985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082: No bootstrap required, opened a new log
I20260620 23:13:36.953550 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:36.947715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.16.79.62:45393)
0620 23:13:36.948079 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:36.948089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:36.948126 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:13:36.948631 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:36.948639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:36.948798 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:36.948994 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:36.949239 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:36.950362 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:36.951587 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:36.952057 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:36.952150 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:36.952835 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:36.952838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:36.952838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:36.952997 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:36.953029 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:36.953039 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:36.953090 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:36.953230 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 23:13:36.953351 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":120,"threads_started":1}
I20260620 23:13:36.953526 25985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e54f1ef93640c99690a339b477f082" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45393 } }
I20260620 23:13:36.953717 25985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:36.953763 25985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e54f1ef93640c99690a339b477f082, State: Initialized, Role: FOLLOWER
I20260620 23:13:36.953869 25985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e54f1ef93640c99690a339b477f082" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45393 } }
I20260620 23:13:36.953974 25985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:36.954017 25985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:36.954064 25985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:36.954660 25985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e54f1ef93640c99690a339b477f082" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45393 } }
I20260620 23:13:36.954780 25985 leader_election.cc:302] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03e54f1ef93640c99690a339b477f082; no voters:
I20260620 23:13:36.954967 25985 leader_election.cc:288] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:36.955022 25990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:36.955243 25990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [term 1 LEADER]: Becoming Leader. State: Replica: 03e54f1ef93640c99690a339b477f082, State: Running, Role: LEADER
I20260620 23:13:36.955246 25985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:36.955351 25990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e54f1ef93640c99690a339b477f082" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45393 } }
I20260620 23:13:36.955773 25992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03e54f1ef93640c99690a339b477f082. Latest consensus state: current_term: 1 leader_uuid: "03e54f1ef93640c99690a339b477f082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e54f1ef93640c99690a339b477f082" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45393 } } }
I20260620 23:13:36.955863 25992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:36.956059 25998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:36.956259 25991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03e54f1ef93640c99690a339b477f082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e54f1ef93640c99690a339b477f082" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45393 } } }
I20260620 23:13:36.956425 25991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:36.956552 25998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:36.961090 25998 catalog_manager.cc:1347] Generated new cluster ID: 6f2f26a1daa04cfe9a8cfd79952f0403
I20260620 23:13:36.961145 25998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:36.975667 25998 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:36.976147 25998 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:36.984432 25998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03e54f1ef93640c99690a339b477f082: Generated new TSK 0
I20260620 23:13:36.984598 25998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:13:36.989822 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45393
--builtin_ntp_servers=127.16.79.20:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 23:13:37.089916 26009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:37.090128 26009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:37.090164 26009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:37.091382 26009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:37.091449 26009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:37.091493 26009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:37.092970 26009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:37.093214 26009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:37.093489 26009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:37.098614 26014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:37.098896 26009 server_base.cc:1034] running on GCE node
W20260620 23:13:37.098634 26017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:37.098620 26015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:37.099431 26009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:37.099710 26009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:37.100911 26009 hybrid_clock.cc:648] HybridClock initialized: now 1781997217100883 us; error 46 us; skew 500 ppm
I20260620 23:13:37.102252 26009 webserver.cc:460] Webserver started at http://127.16.79.1:46637/ using document root <none> and password file <none>
I20260620 23:13:37.102496 26009 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:37.102550 26009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:37.102659 26009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:37.103895 26009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "e8b79c42defb4dccbdba14d2c72f368b"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "f48781e9c94e886742d592294414ce1c"
server_key_iv: "0b97cc07ea3d99f17dc9bd4e88d320ad"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.104256 26009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "e8b79c42defb4dccbdba14d2c72f368b"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "f48781e9c94e886742d592294414ce1c"
server_key_iv: "0b97cc07ea3d99f17dc9bd4e88d320ad"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.105772 26009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:13:37.106585 26023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.106782 26009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:37.106866 26009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "e8b79c42defb4dccbdba14d2c72f368b"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "f48781e9c94e886742d592294414ce1c"
server_key_iv: "0b97cc07ea3d99f17dc9bd4e88d320ad"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.106927 26009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:37.132009 26009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:37.132266 26009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:37.132382 26009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:37.132561 26009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:37.133163 26009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:37.133315 26009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.133374 26009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:37.133431 26009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.140895 26009 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:37031
I20260620 23:13:37.140976 26136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:37031 every 8 connection(s)
I20260620 23:13:37.141306 26009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:37.145879 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 26009
I20260620 23:13:37.146008 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:13:37.146276 16700 external_mini_cluster.cc:1428] Setting key deadabc3e364a24d68ffb8036e3ee436
I20260620 23:13:37.147339 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:37.142055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:44813 (local address 127.16.79.62:45393)
0620 23:13:37.142127 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:37.142130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:37.142453 (+ 323us) server_negotiation.cc:407] Connection header received
0620 23:13:37.142939 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:37.142940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:37.142953 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:37.143018 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:37.143715 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:37.144251 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:37.145332 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:37.145759 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:37.145923 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:37.147014 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:37.147016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:37.147016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:37.147021 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:37.147043 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:37.147048 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:37.147115 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:37.147224 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 23:13:37.147258 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:13:37.147583 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:37.142053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45393 (local address 127.16.79.1:44813)
0620 23:13:37.142309 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:37.142342 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:37.142828 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:37.143090 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:37.143094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:37.143102 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:37.143615 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:37.143619 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:37.144334 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:37.144335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:37.145200 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:37.145202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:37.146068 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:37.146072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:37.146235 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:37.146853 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:37.146853 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:37.146895 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:37.146917 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:37.147127 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:37.147131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:37.147134 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:13:37.147197 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:13:37.147305 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":109,"threads_started":1}
I20260620 23:13:37.147568 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45393
--builtin_ntp_servers=127.16.79.20:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:13:37.148267 26137 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45393
I20260620 23:13:37.148360 26137 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:37.148581 26137 heartbeater.cc:511] Master 127.16.79.62:45393 requested a full tablet report, sending...
I20260620 23:13:37.149223 25949 ts_manager.cc:194] Registered new tserver with Master: e8b79c42defb4dccbdba14d2c72f368b (127.16.79.1:37031)
I20260620 23:13:37.149847 25949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.79.1:44813
W20260620 23:13:37.231101 26140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:37.231287 26140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:37.231307 26140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:37.232527 26140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:37.232564 26140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:37.232589 26140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:37.233943 26140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:37.234167 26140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:37.234455 26140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:37.239670 26148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:37.239705 26145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:37.239813 26140 server_base.cc:1034] running on GCE node
W20260620 23:13:37.239718 26146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:37.240026 26140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:37.240269 26140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:37.241446 26140 hybrid_clock.cc:648] HybridClock initialized: now 1781997217241442 us; error 37 us; skew 500 ppm
I20260620 23:13:37.242848 26140 webserver.cc:460] Webserver started at http://127.16.79.2:41335/ using document root <none> and password file <none>
I20260620 23:13:37.243103 26140 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:37.243178 26140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:37.243376 26140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:37.244469 26140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "cff4d22abebc4ebe9c2248497affe132"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "31d06ae7195085e449161e5e2b813053"
server_key_iv: "065a118f0b5183386d25aeaf1912d0d5"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.244892 26140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "cff4d22abebc4ebe9c2248497affe132"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "31d06ae7195085e449161e5e2b813053"
server_key_iv: "065a118f0b5183386d25aeaf1912d0d5"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.246304 26140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:37.247188 26154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.247370 26140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:37.247493 26140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "cff4d22abebc4ebe9c2248497affe132"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "31d06ae7195085e449161e5e2b813053"
server_key_iv: "065a118f0b5183386d25aeaf1912d0d5"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.247576 26140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:37.260908 26140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:37.261322 26140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:37.261492 26140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:37.261771 26140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:37.262205 26140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:37.262265 26140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.262301 26140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:37.262334 26140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.269644 26140 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:46839
I20260620 23:13:37.269701 26267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:46839 every 8 connection(s)
I20260620 23:13:37.270226 26140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:13:37.272926 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 26140
I20260620 23:13:37.273030 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:37.273239 16700 external_mini_cluster.cc:1428] Setting key 1bfa40cd337aafce633c347401ab1a79
I20260620 23:13:37.274255 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45393
--builtin_ntp_servers=127.16.79.20:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:13:37.276587 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:37.271098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:59685 (local address 127.16.79.62:45393)
0620 23:13:37.271174 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:37.271176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:37.271330 (+ 154us) server_negotiation.cc:407] Connection header received
0620 23:13:37.271847 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:37.271848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:37.271859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:37.271939 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:37.272635 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:37.273540 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:37.274638 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:37.274988 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:37.275067 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:37.276192 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:37.276193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:37.276193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:37.276197 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:37.276210 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:37.276285 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:37.276343 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:37.276505 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 23:13:37.276526 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:13:37.276883 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:37.271022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45393 (local address 127.16.79.2:59685)
0620 23:13:37.271216 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:37.271257 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:37.271735 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:37.272011 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:37.272014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:37.272023 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:37.272532 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:37.272536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:37.273637 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:37.273638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:37.274551 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:37.274554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:37.275078 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:37.275078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:37.275181 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:37.275875 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:37.275876 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:37.275933 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:37.275977 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:37.276371 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:37.276378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:37.276382 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:37.276471 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:13:37.276595 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260620 23:13:37.277886 26268 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45393
I20260620 23:13:37.277989 26268 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:37.278262 26268 heartbeater.cc:511] Master 127.16.79.62:45393 requested a full tablet report, sending...
I20260620 23:13:37.278729 25949 ts_manager.cc:194] Registered new tserver with Master: cff4d22abebc4ebe9c2248497affe132 (127.16.79.2:46839)
I20260620 23:13:37.279266 25949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.79.2:59685
W20260620 23:13:37.359513 26271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:37.359695 26271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:37.359716 26271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:37.360854 26271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:37.360891 26271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:37.360916 26271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:37.362203 26271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:37.362437 26271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:37.362725 26271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:37.367681 26276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:37.367676 26279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:37.367712 26277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:37.367920 26271 server_base.cc:1034] running on GCE node
I20260620 23:13:37.368132 26271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:37.368373 26271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:37.369567 26271 hybrid_clock.cc:648] HybridClock initialized: now 1781997217369549 us; error 29 us; skew 500 ppm
I20260620 23:13:37.370795 26271 webserver.cc:460] Webserver started at http://127.16.79.3:43087/ using document root <none> and password file <none>
I20260620 23:13:37.371052 26271 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:37.371096 26271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:37.371237 26271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:37.372294 26271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "696153529d5c4e76b2bdb47179e1ee80"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "ee80b18fd08341de4eaddf2320b8f0e3"
server_key_iv: "5aaf585548b709e893ce9a38b8cf55da"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.372644 26271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "696153529d5c4e76b2bdb47179e1ee80"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "ee80b18fd08341de4eaddf2320b8f0e3"
server_key_iv: "5aaf585548b709e893ce9a38b8cf55da"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.373992 26271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:37.374964 26285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.375367 26271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:37.375509 26271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "696153529d5c4e76b2bdb47179e1ee80"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "ee80b18fd08341de4eaddf2320b8f0e3"
server_key_iv: "5aaf585548b709e893ce9a38b8cf55da"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.375595 26271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:37.400812 26271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:37.401433 26271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:37.401659 26271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:37.401954 26271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:37.402354 26271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:37.402417 26271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.402465 26271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:37.402501 26271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.410012 26271 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:43779
I20260620 23:13:37.410068 26398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:43779 every 8 connection(s)
I20260620 23:13:37.410485 26271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:13:37.416499 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 26271
I20260620 23:13:37.416595 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:37.416858 16700 external_mini_cluster.cc:1428] Setting key c4aa9ba5faa96bf46487f5090a92dac9
I20260620 23:13:37.416978 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:37.411238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:57447 (local address 127.16.79.62:45393)
0620 23:13:37.411306 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:37.411309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:37.411592 (+ 283us) server_negotiation.cc:407] Connection header received
0620 23:13:37.412247 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:37.412248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:37.412260 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:37.412311 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:37.413196 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:37.413862 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:37.415213 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:37.415440 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:37.415547 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:37.416491 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:37.416493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:37.416493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:37.416499 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:37.416517 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:37.416521 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:37.416659 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:37.416840 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 23:13:37.416875 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:13:37.417305 26401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:37.411221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45393 (local address 127.16.79.3:57447)
0620 23:13:37.411433 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:37.411504 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:37.412131 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:37.412373 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:37.412377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:37.412387 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:37.413106 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:37.413111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:37.413982 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:37.413983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:37.415014 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:37.415016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:37.415560 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:37.415561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:37.415664 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:37.416232 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:37.416233 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:37.416265 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:37.416286 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:37.416671 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:37.416686 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:37.416722 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:13:37.416842 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 23:13:37.417001 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
I20260620 23:13:37.417835 26399 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45393
I20260620 23:13:37.418027 26399 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:37.418229 26399 heartbeater.cc:511] Master 127.16.79.62:45393 requested a full tablet report, sending...
I20260620 23:13:37.418603 25949 ts_manager.cc:194] Registered new tserver with Master: 696153529d5c4e76b2bdb47179e1ee80 (127.16.79.3:43779)
I20260620 23:13:37.418931 25949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.79.3:57447
I20260620 23:13:37.428987 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:13:37.431113 25988 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:37.430519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.16.79.62:45393)
0620 23:13:37.430618 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:37.430620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:37.430735 (+ 115us) server_negotiation.cc:407] Connection header received
0620 23:13:37.430812 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:37.430813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:37.430823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:37.430876 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:37.431018 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34982: BlockingRecv error: recv got EOF from 127.0.0.1:34982 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:13:37.431270 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 26009
I20260620 23:13:37.437052 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 26140
I20260620 23:13:37.441931 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 26271
I20260620 23:13:37.446094 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 25918
2026-06-20T23:13:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (651 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T23:13:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:37Z Disabled control of system clock
I20260620 23:13:37.474666 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45701
--webserver_interface=127.16.79.62
--webserver_port=0
--builtin_ntp_servers=127.16.79.20:32993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:45701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:13:37.556430 26410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:37.556633 26410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:37.556654 26410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:37.557816 26410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:37.557866 26410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:37.557879 26410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:37.557893 26410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:37.557904 26410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:37.559227 26410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:32993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:45701
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:37.559480 26410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:37.559773 26410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:37.564761 26416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:37.564743 26415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:37.564900 26410 server_base.cc:1034] running on GCE node
W20260620 23:13:37.564678 26418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:37.565269 26410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:37.565548 26410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:37.566757 26410 hybrid_clock.cc:648] HybridClock initialized: now 1781997217566739 us; error 25 us; skew 500 ppm
I20260620 23:13:37.568051 26410 webserver.cc:460] Webserver started at http://127.16.79.62:40923/ using document root <none> and password file <none>
I20260620 23:13:37.568293 26410 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:37.568362 26410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:37.568540 26410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:37.569617 26410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "85d04c4119184d7a932cc0877a5c9529"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "a6695057a7b10a9a9adbd85d352afbf9"
server_key_iv: "e64d6f417a593e89e5594d678dbc8fd0"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.569977 26410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "85d04c4119184d7a932cc0877a5c9529"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "a6695057a7b10a9a9adbd85d352afbf9"
server_key_iv: "e64d6f417a593e89e5594d678dbc8fd0"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.571210 26410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:13:37.571965 26424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.572131 26410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.572237 26410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "85d04c4119184d7a932cc0877a5c9529"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "a6695057a7b10a9a9adbd85d352afbf9"
server_key_iv: "e64d6f417a593e89e5594d678dbc8fd0"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.572324 26410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:37.575724 26410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:37.575963 26410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:37.576099 26410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:37.580314 26410 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:45701
I20260620 23:13:37.580338 26476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:45701 every 8 connection(s)
I20260620 23:13:37.580644 26410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:37.581270 26477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:37.583813 26477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529: Bootstrap starting.
I20260620 23:13:37.584389 26477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:37.584739 26477 log.cc:824] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:37.585402 26477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529: No bootstrap required, opened a new log
I20260620 23:13:37.586905 26477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d04c4119184d7a932cc0877a5c9529" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45701 } }
I20260620 23:13:37.587028 26477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:37.587049 26477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d04c4119184d7a932cc0877a5c9529, State: Initialized, Role: FOLLOWER
I20260620 23:13:37.587204 26477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d04c4119184d7a932cc0877a5c9529" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45701 } }
I20260620 23:13:37.587281 26477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:37.587302 26477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:37.587363 26477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:37.587860 26477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d04c4119184d7a932cc0877a5c9529" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45701 } }
I20260620 23:13:37.587939 26477 leader_election.cc:302] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85d04c4119184d7a932cc0877a5c9529; no voters:
I20260620 23:13:37.588102 26477 leader_election.cc:288] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:37.588145 26480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:37.588341 26480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [term 1 LEADER]: Becoming Leader. State: Replica: 85d04c4119184d7a932cc0877a5c9529, State: Running, Role: LEADER
I20260620 23:13:37.588348 26477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:37.588452 26480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d04c4119184d7a932cc0877a5c9529" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45701 } }
I20260620 23:13:37.588900 26481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85d04c4119184d7a932cc0877a5c9529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d04c4119184d7a932cc0877a5c9529" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45701 } } }
I20260620 23:13:37.588989 26481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:37.589411 26482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85d04c4119184d7a932cc0877a5c9529. Latest consensus state: current_term: 1 leader_uuid: "85d04c4119184d7a932cc0877a5c9529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d04c4119184d7a932cc0877a5c9529" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45701 } } }
I20260620 23:13:37.589499 26482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:37.589403 26489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:37.589948 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 26410
I20260620 23:13:37.590015 26489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:37.590039 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:37.590258 16700 external_mini_cluster.cc:1428] Setting key 8c437a7d8d9b20b0b0f1f2771f00d1d3
I20260620 23:13:37.591586 26489 catalog_manager.cc:1347] Generated new cluster ID: 9d22a77118ee411abc9567cbcb800cb2
I20260620 23:13:37.591638 26489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:37.594653 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:37.590662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.16.79.62:45701)
0620 23:13:37.590849 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:37.590858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:37.592932 (+ 2074us) server_negotiation.cc:407] Connection header received
0620 23:13:37.593062 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:37.593068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:37.593110 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:37.593247 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:37.593881 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:37.593884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:37.593885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:37.593888 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:37.593928 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:37.593940 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:37.593997 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:37.594087 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 23:13:37.594309 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260620 23:13:37.651731 26489 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:37.651940 26489 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:37.657253 26489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85d04c4119184d7a932cc0877a5c9529: Generated new TSK 0
I20260620 23:13:37.657445 26489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:13:37.666438 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45701
--builtin_ntp_servers=127.16.79.20:32993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:13:37.764192 26501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:37.764385 26501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:37.764407 26501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:37.765571 26501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:37.765610 26501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:37.765635 26501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:37.767292 26501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:32993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:37.767530 26501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:37.767915 26501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:37.772943 26507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:37.772972 26509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:37.772953 26506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:37.773170 26501 server_base.cc:1034] running on GCE node
I20260620 23:13:37.773429 26501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:37.773651 26501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:37.774793 26501 hybrid_clock.cc:648] HybridClock initialized: now 1781997217774765 us; error 65 us; skew 500 ppm
I20260620 23:13:37.776204 26501 webserver.cc:460] Webserver started at http://127.16.79.1:40885/ using document root <none> and password file <none>
I20260620 23:13:37.776449 26501 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:37.776515 26501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:37.776875 26501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:37.778473 26501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "92e246298b134188ab370c636c960357"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "546a19b2f2b1a37b45faadfb1026c0f9"
server_key_iv: "8b41670954d05946646d0fdcd13117da"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.778929 26501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "92e246298b134188ab370c636c960357"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "546a19b2f2b1a37b45faadfb1026c0f9"
server_key_iv: "8b41670954d05946646d0fdcd13117da"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.780591 26501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:13:37.781589 26515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.781945 26501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:37.782127 26501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "92e246298b134188ab370c636c960357"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "546a19b2f2b1a37b45faadfb1026c0f9"
server_key_iv: "8b41670954d05946646d0fdcd13117da"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.782279 26501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:37.785640 26501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:37.785902 26501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:37.786041 26501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:37.786257 26501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:37.786550 26501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:37.786602 26501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.786641 26501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:37.786669 26501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.792497 26501 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:45803
I20260620 23:13:37.792625 26628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:45803 every 8 connection(s)
I20260620 23:13:37.792974 26501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:37.796921 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:37.793827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:60875 (local address 127.16.79.62:45701)
0620 23:13:37.793910 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:37.793913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:37.794174 (+ 261us) server_negotiation.cc:407] Connection header received
0620 23:13:37.794904 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:37.794906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:37.794917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:37.794985 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:37.796402 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:37.796403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:37.796404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:37.796409 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:37.796428 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:37.796434 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:37.796500 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:37.796688 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 23:13:37.796764 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:13:37.796938 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:37.793766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45701 (local address 127.16.79.1:60875)
0620 23:13:37.794000 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:37.794044 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:37.794752 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:37.795058 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:37.795062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:37.795072 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:37.796152 (+ 1080us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:37.796153 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:37.796181 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:37.796226 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:37.796538 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:37.796543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:37.796546 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:13:37.796639 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:13:37.796773 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260620 23:13:37.797732 26629 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45701
I20260620 23:13:37.797822 26629 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:37.797925 26629 heartbeater.cc:511] Master 127.16.79.62:45701 requested a full tablet report, sending...
I20260620 23:13:37.798904 26441 ts_manager.cc:194] Registered new tserver with Master: 92e246298b134188ab370c636c960357 (127.16.79.1:45803)
I20260620 23:13:37.802695 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 26501
I20260620 23:13:37.802776 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:13:37.803002 16700 external_mini_cluster.cc:1428] Setting key 7e403398d89b89516fd087d13a0cead3
I20260620 23:13:37.803956 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45701
--builtin_ntp_servers=127.16.79.20:32993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:13:37.885406 26632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:37.885681 26632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:37.885741 26632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:37.887673 26632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:37.887742 26632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:37.887794 26632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:37.889815 26632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:32993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:37.890018 26632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:37.890307 26632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:37.895623 26640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:37.895608 26637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:37.895740 26638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:37.895738 26632 server_base.cc:1034] running on GCE node
I20260620 23:13:37.896131 26632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:37.896384 26632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:37.897569 26632 hybrid_clock.cc:648] HybridClock initialized: now 1781997217897547 us; error 52 us; skew 500 ppm
I20260620 23:13:37.899075 26632 webserver.cc:460] Webserver started at http://127.16.79.2:43533/ using document root <none> and password file <none>
I20260620 23:13:37.899415 26632 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:37.899613 26632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:37.899791 26632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:37.900947 26632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "0fd339b16fff422baf22d6dade9cbc90"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "2579736f43a3ebabbe1251f88feb4f6b"
server_key_iv: "fe6998d8651efb2eac76f1d0785ad6ce"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.901304 26632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "0fd339b16fff422baf22d6dade9cbc90"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "2579736f43a3ebabbe1251f88feb4f6b"
server_key_iv: "fe6998d8651efb2eac76f1d0785ad6ce"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.902913 26632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:13:37.903795 26646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.903986 26632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:37.904083 26632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "0fd339b16fff422baf22d6dade9cbc90"
format_stamp: "Formatted at 2026-06-20 23:13:37 on dist-test-slave-pswb"
server_key: "2579736f43a3ebabbe1251f88feb4f6b"
server_key_iv: "fe6998d8651efb2eac76f1d0785ad6ce"
server_key_version: "encryptionkey@0"
I20260620 23:13:37.904160 26632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:37.907647 26632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:37.907898 26632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:37.908041 26632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:37.908258 26632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:37.908598 26632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:37.908650 26632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.908715 26632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:37.908743 26632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:37.914650 26632 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:35099
I20260620 23:13:37.914702 26759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:35099 every 8 connection(s)
I20260620 23:13:37.915069 26632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:13:37.918179 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:37.915815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:42967 (local address 127.16.79.62:45701)
0620 23:13:37.915956 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:37.915958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:37.916138 (+ 180us) server_negotiation.cc:407] Connection header received
0620 23:13:37.916650 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:37.916651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:37.916657 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:37.916723 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:37.917899 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:37.917900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:37.917900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:37.917905 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:37.917921 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:37.917925 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:37.917995 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:37.918083 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 23:13:37.918098 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 23:13:37.918354 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:37.915783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45701 (local address 127.16.79.2:42967)
0620 23:13:37.916006 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:37.916052 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:37.916541 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:37.916838 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:37.916842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:37.916849 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:37.917737 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:37.917737 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:37.917763 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:37.917790 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:37.918007 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:37.918013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:37.918015 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:13:37.918092 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 23:13:37.918149 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260620 23:13:37.918728 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 26632
I20260620 23:13:37.918776 26760 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45701
I20260620 23:13:37.918848 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:37.918850 26760 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:37.918990 26760 heartbeater.cc:511] Master 127.16.79.62:45701 requested a full tablet report, sending...
I20260620 23:13:37.919059 16700 external_mini_cluster.cc:1428] Setting key 0f5359456989c18194387bd2a5c16541
I20260620 23:13:37.919327 26441 ts_manager.cc:194] Registered new tserver with Master: 0fd339b16fff422baf22d6dade9cbc90 (127.16.79.2:35099)
I20260620 23:13:37.920166 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45701
--builtin_ntp_servers=127.16.79.20:32993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:13:37.999766 26763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:37.999960 26763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:37.999982 26763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:38.001281 26763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:38.001366 26763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:38.001405 26763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:38.003376 26763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:32993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:38.003652 26763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:38.003962 26763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:38.010519 26769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:38.011155 26763 server_base.cc:1034] running on GCE node
W20260620 23:13:38.011180 26768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:38.011183 26771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:38.011613 26763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:38.011898 26763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:38.013073 26763 hybrid_clock.cc:648] HybridClock initialized: now 1781997218013046 us; error 79 us; skew 500 ppm
I20260620 23:13:38.014317 26763 webserver.cc:460] Webserver started at http://127.16.79.3:45289/ using document root <none> and password file <none>
I20260620 23:13:38.014549 26763 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:38.014613 26763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:38.014775 26763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:38.015784 26763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "87b8f6b9f88144db9b6f0237f396b2cc"
format_stamp: "Formatted at 2026-06-20 23:13:38 on dist-test-slave-pswb"
server_key: "8600b1941ae2c4e250933e4374781fa7"
server_key_iv: "279b717a9252e96d806212f91e344fc6"
server_key_version: "encryptionkey@0"
I20260620 23:13:38.016129 26763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "87b8f6b9f88144db9b6f0237f396b2cc"
format_stamp: "Formatted at 2026-06-20 23:13:38 on dist-test-slave-pswb"
server_key: "8600b1941ae2c4e250933e4374781fa7"
server_key_iv: "279b717a9252e96d806212f91e344fc6"
server_key_version: "encryptionkey@0"
I20260620 23:13:38.017568 26763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:13:38.018364 26777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:38.018587 26763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:38.018688 26763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "87b8f6b9f88144db9b6f0237f396b2cc"
format_stamp: "Formatted at 2026-06-20 23:13:38 on dist-test-slave-pswb"
server_key: "8600b1941ae2c4e250933e4374781fa7"
server_key_iv: "279b717a9252e96d806212f91e344fc6"
server_key_version: "encryptionkey@0"
I20260620 23:13:38.018764 26763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:38.022173 26763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:38.022445 26763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:38.022580 26763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:38.022804 26763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:38.023109 26763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:38.023164 26763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:38.023202 26763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:38.023228 26763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:38.028837 26763 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:33131
I20260620 23:13:38.028869 26890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:33131 every 8 connection(s)
I20260620 23:13:38.029161 26763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:13:38.031782 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:38.029798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:55853 (local address 127.16.79.62:45701)
0620 23:13:38.029839 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:38.029841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:38.030043 (+ 202us) server_negotiation.cc:407] Connection header received
0620 23:13:38.030494 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:38.030495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:38.030500 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:38.030560 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:38.031583 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:38.031584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:38.031584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:13:38.031588 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:38.031600 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:13:38.031603 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:38.031666 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:38.031712 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 23:13:38.031728 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:13:38.031970 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:38.029756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45701 (local address 127.16.79.3:55853)
0620 23:13:38.029932 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:38.029970 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:38.030385 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:38.030625 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:38.030628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:38.030635 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:38.031448 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:13:38.031449 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:38.031473 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:13:38.031497 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:38.031665 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:38.031667 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:38.031669 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:13:38.031717 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 23:13:38.031758 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260620 23:13:38.032616 26891 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45701
I20260620 23:13:38.032784 26891 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:38.032934 26891 heartbeater.cc:511] Master 127.16.79.62:45701 requested a full tablet report, sending...
I20260620 23:13:38.033367 26440 ts_manager.cc:194] Registered new tserver with Master: 87b8f6b9f88144db9b6f0237f396b2cc (127.16.79.3:33131)
I20260620 23:13:38.035123 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 26763
I20260620 23:13:38.035197 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:38.035339 16700 external_mini_cluster.cc:1428] Setting key ac2a9bbe30c8eec87ab914695e52358d
I20260620 23:13:38.036221 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:13:38.038445 26499 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:13:38.037789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.16.79.62:45701)
0620 23:13:38.037871 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:38.037873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:38.037967 (+ 94us) server_negotiation.cc:407] Connection header received
0620 23:13:38.038056 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:38.038057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:38.038060 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:38.038104 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:38.038394 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55812: BlockingRecv error: recv got EOF from 127.0.0.1:55812 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:13:38.038654 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 26501
I20260620 23:13:38.043644 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 26632
I20260620 23:13:38.047822 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 26763
I20260620 23:13:38.053755 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 26410
2026-06-20T23:13:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (607 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:13:38 dist-test-slave-pswb krb5kdc[26901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:13:38 dist-test-slave-pswb krb5kdc[26901](info): set up 2 sockets
Jun 20 23:13:38 dist-test-slave-pswb krb5kdc[26901](info): commencing operation
krb5kdc: starting...
W20260620 23:13:39.098541 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:13:39 dist-test-slave-pswb krb5kdc[26901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:13:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:13:39.219885 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:46155
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:46155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:39.299435 26914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:39.299640 26914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:39.299674 26914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:39.300805 26914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:39.300858 26914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:39.300882 26914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:39.300911 26914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:39.300938 26914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:39.302335 26914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:46155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:46155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:39.302579 26914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:39.302788 26914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:39.307518 26920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:39.307590 26919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:39.307632 26914 server_base.cc:1034] running on GCE node
W20260620 23:13:39.307711 26922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:39.307926 26914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:39.308187 26914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:39.309403 26914 hybrid_clock.cc:648] HybridClock initialized: now 1781997219309381 us; error 43 us; skew 500 ppm
Jun 20 23:13:39 dist-test-slave-pswb krb5kdc[26901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:39.311765 26914 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:13:39.312165 26914 webserver.cc:460] Webserver started at http://127.16.79.62:40289/ using document root <none> and password file <none>
I20260620 23:13:39.312400 26914 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:39.312466 26914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:39.312614 26914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:39.313603 26914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "f0a85cfb65fc4b5e867d783d21124621"
format_stamp: "Formatted at 2026-06-20 23:13:39 on dist-test-slave-pswb"
server_key: "a2ee643e348a79210b1674c4f7094d34"
server_key_iv: "43436be99d6d626eb780b28219addc96"
server_key_version: "encryptionkey@0"
I20260620 23:13:39.313963 26914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "f0a85cfb65fc4b5e867d783d21124621"
format_stamp: "Formatted at 2026-06-20 23:13:39 on dist-test-slave-pswb"
server_key: "a2ee643e348a79210b1674c4f7094d34"
server_key_iv: "43436be99d6d626eb780b28219addc96"
server_key_version: "encryptionkey@0"
I20260620 23:13:39.315220 26914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:13:39.316237 26929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:39.316469 26914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:39.316578 26914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "f0a85cfb65fc4b5e867d783d21124621"
format_stamp: "Formatted at 2026-06-20 23:13:39 on dist-test-slave-pswb"
server_key: "a2ee643e348a79210b1674c4f7094d34"
server_key_iv: "43436be99d6d626eb780b28219addc96"
server_key_version: "encryptionkey@0"
I20260620 23:13:39.316664 26914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:39.334386 26914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:39.336388 26914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:39.336546 26914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:39.340271 26914 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:46155
I20260620 23:13:39.340318 26981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:46155 every 8 connection(s)
I20260620 23:13:39.340667 26914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:39.342100 26982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:39.344170 26982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621: Bootstrap starting.
I20260620 23:13:39.344508 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 26914
I20260620 23:13:39.344600 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:13:39.344825 16700 external_mini_cluster.cc:1428] Setting key 88c44e141ea0530b213c5eeedd23671e
I20260620 23:13:39.344882 26982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:39.345187 26982 log.cc:824] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:39.345979 26982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621: No bootstrap required, opened a new log
I20260620 23:13:39.348177 26982 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a85cfb65fc4b5e867d783d21124621" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46155 } }
I20260620 23:13:39.348330 26982 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:39.348369 26982 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a85cfb65fc4b5e867d783d21124621, State: Initialized, Role: FOLLOWER
I20260620 23:13:39.348488 26982 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a85cfb65fc4b5e867d783d21124621" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46155 } }
I20260620 23:13:39.348583 26982 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:39.348606 26982 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:39.348670 26982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:39.349224 26982 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a85cfb65fc4b5e867d783d21124621" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46155 } }
I20260620 23:13:39.349323 26982 leader_election.cc:302] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0a85cfb65fc4b5e867d783d21124621; no voters:
I20260620 23:13:39.349457 26982 leader_election.cc:288] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:39.349536 26987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:39.349664 26982 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:39.349682 26987 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [term 1 LEADER]: Becoming Leader. State: Replica: f0a85cfb65fc4b5e867d783d21124621, State: Running, Role: LEADER
I20260620 23:13:39.349900 26987 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a85cfb65fc4b5e867d783d21124621" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46155 } }
I20260620 23:13:39.350200 26987 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0a85cfb65fc4b5e867d783d21124621. Latest consensus state: current_term: 1 leader_uuid: "f0a85cfb65fc4b5e867d783d21124621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a85cfb65fc4b5e867d783d21124621" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46155 } } }
I20260620 23:13:39.350425 26987 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:39.350347 26989 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0a85cfb65fc4b5e867d783d21124621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a85cfb65fc4b5e867d783d21124621" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46155 } } }
I20260620 23:13:39.350540 26989 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:39.350925 26996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:39.351380 26996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:13:39 dist-test-slave-pswb krb5kdc[26901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:39.353966 26996 catalog_manager.cc:1347] Generated new cluster ID: 3c404161397347609a3f35b9b9635329
I20260620 23:13:39.354023 26996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:39.356276 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.345347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.16.79.62:46155)
0620 23:13:39.345517 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:39.345525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:39.345560 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:13:39.346129 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:39.346137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:39.346191 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:39.346322 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:39.346902 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.348296 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.349248 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.349852 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.350282 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.352750 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:39.352753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:39.352753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:39.352764 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:39.354784 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:39.355161 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:39.355163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:39.355164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:39.355257 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:39.355471 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:39.355471 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:39.355471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:39.355725 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:39.355859 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:39.355972 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 23:13:39.356088 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260620 23:13:39.362821 26996 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:39.363417 26996 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:39.368983 26996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621: Generated new TSK 0
I20260620 23:13:39.369164 26996 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:13:39.395740 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46155
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:39.482280 27008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:39.482550 27008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:39.482579 27008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:39.483806 27008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:39.483850 27008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:39.483878 27008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:13:39.485474 27008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:39.485708 27008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:39.485940 27008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:39.491077 27014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:39.491143 27013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:39.491209 27008 server_base.cc:1034] running on GCE node
W20260620 23:13:39.491092 27016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:39.491495 27008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:39.491725 27008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:39.492892 27008 hybrid_clock.cc:648] HybridClock initialized: now 1781997219492892 us; error 43 us; skew 500 ppm
Jun 20 23:13:39 dist-test-slave-pswb krb5kdc[26901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:39.495417 27008 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:13:39.495904 27008 webserver.cc:460] Webserver started at http://127.16.79.1:33697/ using document root <none> and password file <none>
I20260620 23:13:39.496126 27008 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:39.496194 27008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:39.496343 27008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:39.497385 27008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38"
format_stamp: "Formatted at 2026-06-20 23:13:39 on dist-test-slave-pswb"
server_key: "db2d943ec8ef468434b2d3613bfb954a"
server_key_iv: "daa6de4c307fae8cc44e193adf2a8046"
server_key_version: "encryptionkey@0"
I20260620 23:13:39.497968 27008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38"
format_stamp: "Formatted at 2026-06-20 23:13:39 on dist-test-slave-pswb"
server_key: "db2d943ec8ef468434b2d3613bfb954a"
server_key_iv: "daa6de4c307fae8cc44e193adf2a8046"
server_key_version: "encryptionkey@0"
I20260620 23:13:39.499748 27008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:13:39.500654 27023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:39.500881 27008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:13:39.500962 27008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38"
format_stamp: "Formatted at 2026-06-20 23:13:39 on dist-test-slave-pswb"
server_key: "db2d943ec8ef468434b2d3613bfb954a"
server_key_iv: "daa6de4c307fae8cc44e193adf2a8046"
server_key_version: "encryptionkey@0"
I20260620 23:13:39.501017 27008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:39.516004 27008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:39.518682 27008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:39.518795 27008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:39.518987 27008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:39.519230 27008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:39.519260 27008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:39.519281 27008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:39.519299 27008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:39.525259 27008 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:44977
I20260620 23:13:39.525331 27136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:44977 every 8 connection(s)
I20260620 23:13:39.525646 27008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:13:39 dist-test-slave-pswb krb5kdc[26901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:39.533212 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 27008
I20260620 23:13:39.533288 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:13:39.533484 16700 external_mini_cluster.cc:1428] Setting key f107be14e2c56cae1e98f94b11d1bf60
I20260620 23:13:39.535175 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.526387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:57465 (local address 127.16.79.62:46155)
0620 23:13:39.526458 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:39.526460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:39.526705 (+ 245us) server_negotiation.cc:407] Connection header received
0620 23:13:39.527155 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:39.527156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:39.527166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:39.527219 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:39.528094 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.528581 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.529589 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.529839 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.529897 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.532039 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:39.532040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:39.532041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:39.532046 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:39.533582 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:39.534018 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:39.534019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:39.534020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:39.534083 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:39.534455 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:39.534456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:39.534456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:39.534593 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:39.534664 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:39.535028 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 23:13:39.535096 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:13:39.535462 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.526354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46155 (local address 127.16.79.1:57465)
0620 23:13:39.526591 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:39.526624 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:39.527070 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:39.527265 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:39.527268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:39.527807 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:39.527991 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:39.527995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.528723 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.528723 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:39.529499 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:39.529501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.529933 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.529934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:39.530035 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.530575 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:39.530583 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:39.531938 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:39.533709 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:39.533714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:39.533716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:39.533927 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:39.534201 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:39.534202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:39.534202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:39.534352 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:39.534674 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:39.534679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:39.534855 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 23:13:39.534986 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 23:13:39.535176 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":123,"threads_started":1}
I20260620 23:13:39.535948 27137 heartbeater.cc:347] Connected to a master server at 127.16.79.62:46155
I20260620 23:13:39.536060 27137 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:39.536295 27137 heartbeater.cc:511] Master 127.16.79.62:46155 requested a full tablet report, sending...
I20260620 23:13:39.536856 26946 ts_manager.cc:194] Registered new tserver with Master: 6b1a1a5ef31944f28cb03dd8ccd8da38 (127.16.79.1:44977)
I20260620 23:13:39.537506 26946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:57465
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:13:39.559973 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46155
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:39.639415 27142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:39.639595 27142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:39.639616 27142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:39.640643 27142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:39.640678 27142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:39.640766 27142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:13:39.642119 27142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:39.642287 27142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:39.642439 27142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:39.647255 27150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:39.647290 27148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:39.647280 27147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:39.647411 27142 server_base.cc:1034] running on GCE node
I20260620 23:13:39.647634 27142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:39.647820 27142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:39.648952 27142 hybrid_clock.cc:648] HybridClock initialized: now 1781997219648941 us; error 25 us; skew 500 ppm
Jun 20 23:13:39 dist-test-slave-pswb krb5kdc[26901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:39.651556 27142 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:13:39.651998 27142 webserver.cc:460] Webserver started at http://127.16.79.2:34109/ using document root <none> and password file <none>
I20260620 23:13:39.652215 27142 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:39.652261 27142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:39.652396 27142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:39.653386 27142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "18c6158b53b340ab8ebf10e6a39a80d7"
format_stamp: "Formatted at 2026-06-20 23:13:39 on dist-test-slave-pswb"
server_key: "7464c35ea5b99fd439a722dc1c7f6ebf"
server_key_iv: "54596675721e61f6d751c120010d184b"
server_key_version: "encryptionkey@0"
I20260620 23:13:39.653754 27142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "18c6158b53b340ab8ebf10e6a39a80d7"
format_stamp: "Formatted at 2026-06-20 23:13:39 on dist-test-slave-pswb"
server_key: "7464c35ea5b99fd439a722dc1c7f6ebf"
server_key_iv: "54596675721e61f6d751c120010d184b"
server_key_version: "encryptionkey@0"
I20260620 23:13:39.655061 27142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:13:39.655735 27157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:39.655954 27142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:39.656023 27142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "18c6158b53b340ab8ebf10e6a39a80d7"
format_stamp: "Formatted at 2026-06-20 23:13:39 on dist-test-slave-pswb"
server_key: "7464c35ea5b99fd439a722dc1c7f6ebf"
server_key_iv: "54596675721e61f6d751c120010d184b"
server_key_version: "encryptionkey@0"
I20260620 23:13:39.656106 27142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:39.677748 27142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:39.679925 27142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:39.680068 27142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:39.680290 27142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:39.680583 27142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:39.680641 27142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:39.680681 27142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:39.680701 27142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:39.686276 27142 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:38429
I20260620 23:13:39.686348 27270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:38429 every 8 connection(s)
I20260620 23:13:39.686637 27142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:13:39 dist-test-slave-pswb krb5kdc[26901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:39.694857 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 27142
I20260620 23:13:39.694983 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:39.695189 16700 external_mini_cluster.cc:1428] Setting key 5e4ee9748f93b5fe138d08f636554495
I20260620 23:13:39.696504 27273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.687329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46155 (local address 127.16.79.2:55635)
0620 23:13:39.687487 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:39.687518 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:39.687992 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:39.688219 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:39.688223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:39.688769 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:39.688937 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:39.688941 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.689614 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.689615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:39.690371 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:39.690373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.690765 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.690767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:39.690883 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.691351 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:39.691359 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:39.693152 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:39.695009 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:39.695015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:39.695017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:39.695337 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:39.695569 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:39.695570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:39.695570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:39.695691 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:39.696003 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:39.696008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:39.696112 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 23:13:39.696247 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 23:13:39.696360 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260620 23:13:39.696527 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.687346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:55635 (local address 127.16.79.62:46155)
0620 23:13:39.687388 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:39.687390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:39.687608 (+ 218us) server_negotiation.cc:407] Connection header received
0620 23:13:39.688094 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:39.688096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:39.688106 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:39.688161 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:39.689033 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.689524 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.690461 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.690670 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.690759 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.693286 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:39.693287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:39.693287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:39.693294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:39.694861 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:39.695421 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:39.695422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:39.695423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:39.695490 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:39.695769 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:39.695770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:39.695771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:39.695926 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:39.696009 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:39.696275 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:13:39.696371 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:13:39.697281 27271 heartbeater.cc:347] Connected to a master server at 127.16.79.62:46155
I20260620 23:13:39.697521 27271 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:39.697872 27271 heartbeater.cc:511] Master 127.16.79.62:46155 requested a full tablet report, sending...
I20260620 23:13:39.698540 26946 ts_manager.cc:194] Registered new tserver with Master: 18c6158b53b340ab8ebf10e6a39a80d7 (127.16.79.2:38429)
I20260620 23:13:39.698974 26946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:55635
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:13:39.723150 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46155
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:39.802935 27276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:39.803108 27276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:39.803131 27276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:39.804308 27276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:39.804347 27276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:39.804373 27276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:13:39.805778 27276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:13:39.806007 27276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:39.806228 27276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:39.811208 27282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:39.811214 27281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:39.811259 27284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:39.811450 27276 server_base.cc:1034] running on GCE node
I20260620 23:13:39.811657 27276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:39.811879 27276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:39.813119 27276 hybrid_clock.cc:648] HybridClock initialized: now 1781997219813094 us; error 36 us; skew 500 ppm
Jun 20 23:13:39 dist-test-slave-pswb krb5kdc[26901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:39.816013 27276 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:13:39.816475 27276 webserver.cc:460] Webserver started at http://127.16.79.3:38037/ using document root <none> and password file <none>
I20260620 23:13:39.816843 27276 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:39.816939 27276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:39.817086 27276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:13:39.818096 27276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "596a7e2f307a4854b21a3ce8a4591e2f"
format_stamp: "Formatted at 2026-06-20 23:13:39 on dist-test-slave-pswb"
server_key: "5fcd2c6b4dadd5b64980457bda380fa6"
server_key_iv: "7cb8c5ca19b0b84222e0affbe7ba0a24"
server_key_version: "encryptionkey@0"
I20260620 23:13:39.818460 27276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "596a7e2f307a4854b21a3ce8a4591e2f"
format_stamp: "Formatted at 2026-06-20 23:13:39 on dist-test-slave-pswb"
server_key: "5fcd2c6b4dadd5b64980457bda380fa6"
server_key_iv: "7cb8c5ca19b0b84222e0affbe7ba0a24"
server_key_version: "encryptionkey@0"
I20260620 23:13:39.819762 27276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:13:39.820580 27291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:39.820775 27276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:13:39.820892 27276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "596a7e2f307a4854b21a3ce8a4591e2f"
format_stamp: "Formatted at 2026-06-20 23:13:39 on dist-test-slave-pswb"
server_key: "5fcd2c6b4dadd5b64980457bda380fa6"
server_key_iv: "7cb8c5ca19b0b84222e0affbe7ba0a24"
server_key_version: "encryptionkey@0"
I20260620 23:13:39.820978 27276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:39.847184 27276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:39.849586 27276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:39.849737 27276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:39.849977 27276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:39.850346 27276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:39.850406 27276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:39.850445 27276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:39.850471 27276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:39.857264 27276 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:38079
I20260620 23:13:39.857445 27404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:38079 every 8 connection(s)
I20260620 23:13:39.857687 27276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:13:39.859395 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 27276
I20260620 23:13:39.859508 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:39.859719 16700 external_mini_cluster.cc:1428] Setting key 75e706416787ff9c63aa6f51f012258c
Jun 20 23:13:39 dist-test-slave-pswb krb5kdc[26901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:13:39.868093 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.858753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:57223 (local address 127.16.79.62:46155)
0620 23:13:39.858813 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:39.858815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:39.858987 (+ 172us) server_negotiation.cc:407] Connection header received
0620 23:13:39.859461 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:39.859463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:39.859476 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:39.859555 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:39.860974 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.861659 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.862921 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.863122 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.863186 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.865497 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:39.865498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:39.865498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:39.865505 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:39.866731 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:39.867147 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:39.867148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:39.867148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:39.867200 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:39.867508 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:39.867509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:39.867509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:39.867625 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:39.867686 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:39.867952 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:13:39.868031 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:13:39.868249 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.858648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46155 (local address 127.16.79.3:57223)
0620 23:13:39.858842 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:39.858882 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:39.859354 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:39.859710 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:39.859714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:39.860589 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:39.860854 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:39.860859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.861795 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.861797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:39.862823 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:39.862828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.863223 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.863224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:39.863323 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.863854 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:39.863863 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:39.865371 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:39.866821 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:39.866828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:39.866829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:39.867041 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:39.867284 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:39.867285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:39.867286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:39.867406 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:39.867705 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:39.867710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:39.867822 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 23:13:39.867914 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:13:39.868033 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260620 23:13:39.868896 27405 heartbeater.cc:347] Connected to a master server at 127.16.79.62:46155
I20260620 23:13:39.869028 27405 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:39.869223 27405 heartbeater.cc:511] Master 127.16.79.62:46155 requested a full tablet report, sending...
I20260620 23:13:39.869616 26946 ts_manager.cc:194] Registered new tserver with Master: 596a7e2f307a4854b21a3ce8a4591e2f (127.16.79.3:38079)
I20260620 23:13:39.869972 26946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:57223
I20260620 23:13:39.871594 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:13:39.880167 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.873067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.16.79.62:46155)
0620 23:13:39.873136 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:39.873138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:39.873145 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:13:39.873163 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:39.873164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:39.873179 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:39.873240 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:39.873787 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.874251 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.874968 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.875168 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.875251 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.876448 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:39.876450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:39.876450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:39.876457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:39.878448 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:39.878857 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:39.878858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:39.878859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:39.878997 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:39.879392 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:39.879393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:39.879394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:39.879588 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:39.879846 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:39.879985 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 23:13:39.880068 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:13:39.881855 26946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:13:39.883365 26946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:13:39.890455 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.885838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:44977 (local address 127.0.0.1:41874)
0620 23:13:39.886280 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:39.886292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:39.886335 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:39.886842 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:39.886843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:39.886846 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:39.886899 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:39.886901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.888389 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.888390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:39.889393 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:39.889395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.890110 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.890111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:39.890194 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.890207 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:13:39.890315 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:13:39.890376 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":14,"thread_start_us":132,"threads_started":1}
I20260620 23:13:39.890465 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.886471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.16.79.1:44977)
0620 23:13:39.886677 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:39.886681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:39.886691 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:13:39.886726 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:39.886728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:39.886743 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:39.886969 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:39.887027 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.888278 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.889497 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.890015 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.890108 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.890169 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:39.890333 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 23:13:39.890375 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260620 23:13:39.890808 27417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.885838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:38429 (local address 127.0.0.1:39074)
0620 23:13:39.886139 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:39.886149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:39.886192 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:39.886568 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:39.886571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:39.886573 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:39.886636 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:39.886638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.888219 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.888220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:39.889414 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:39.889416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.890404 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.890405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:39.890475 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.890482 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:13:39.890515 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 23:13:39.890585 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":32,"thread_start_us":148,"threads_started":1}
I20260620 23:13:39.890808 27418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.885819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:38079 (local address 127.0.0.1:49996)
0620 23:13:39.886075 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:39.886116 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:39.886184 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:39.886803 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:39.886804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:39.886807 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:39.886869 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:39.886871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.887952 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.887953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:39.889413 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:39.889416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.890419 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.890420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:39.890475 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.890479 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:39.890515 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 23:13:39.890586 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260620 23:13:39.891196 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.886260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.16.79.3:38079)
0620 23:13:39.886468 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:39.886475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:39.886490 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:13:39.886637 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:39.886641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:39.886674 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:39.886950 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:39.887019 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.887844 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.889523 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.890289 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.890835 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.891049 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:39.891076 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 23:13:39.891119 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260620 23:13:39.891211 27419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:39.886171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.16.79.2:38429)
0620 23:13:39.886381 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:39.886387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:39.886397 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:13:39.886437 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:39.886439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:39.886455 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:39.886716 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:39.886789 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.888104 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.889523 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:39.890289 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:39.891019 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:39.891075 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:39.891098 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 23:13:39.891135 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":95,"threads_started":1}
I20260620 23:13:39.891314 27071 tablet_service.cc:1467] Processing CreateTablet for tablet 619a71c834d84e82b940f7facecefa2f (DEFAULT_TABLE table=test-table [id=3b3a382713c845d59ceee45c8058ebe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:39.891691 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619a71c834d84e82b940f7facecefa2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:39.891777 27205 tablet_service.cc:1467] Processing CreateTablet for tablet 619a71c834d84e82b940f7facecefa2f (DEFAULT_TABLE table=test-table [id=3b3a382713c845d59ceee45c8058ebe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:39.891880 27339 tablet_service.cc:1467] Processing CreateTablet for tablet 619a71c834d84e82b940f7facecefa2f (DEFAULT_TABLE table=test-table [id=3b3a382713c845d59ceee45c8058ebe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:39.892122 27205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619a71c834d84e82b940f7facecefa2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:39.892211 27339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619a71c834d84e82b940f7facecefa2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:39.894694 27422 tablet_bootstrap.cc:492] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7: Bootstrap starting.
I20260620 23:13:39.895226 27423 tablet_bootstrap.cc:492] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38: Bootstrap starting.
I20260620 23:13:39.895434 27422 tablet_bootstrap.cc:654] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:39.895752 27424 tablet_bootstrap.cc:492] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f: Bootstrap starting.
I20260620 23:13:39.896004 27423 tablet_bootstrap.cc:654] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:39.896042 27422 log.cc:824] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:39.896410 27424 tablet_bootstrap.cc:654] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:39.896610 27423 log.cc:824] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:39.896858 27422 tablet_bootstrap.cc:492] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7: No bootstrap required, opened a new log
I20260620 23:13:39.896955 27422 ts_tablet_manager.cc:1397] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:13:39.897068 27424 log.cc:824] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:39.897420 27423 tablet_bootstrap.cc:492] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38: No bootstrap required, opened a new log
I20260620 23:13:39.897488 27423 ts_tablet_manager.cc:1397] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 23:13:39.897838 27424 tablet_bootstrap.cc:492] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f: No bootstrap required, opened a new log
I20260620 23:13:39.897928 27424 ts_tablet_manager.cc:1397] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:13:39.899230 27422 raft_consensus.cc:348] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6158b53b340ab8ebf10e6a39a80d7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38429 } } peers { permanent_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44977 } } peers { permanent_uuid: "596a7e2f307a4854b21a3ce8a4591e2f" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38079 } }
I20260620 23:13:39.899432 27422 raft_consensus.cc:374] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:39.899468 27422 raft_consensus.cc:729] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6158b53b340ab8ebf10e6a39a80d7, State: Initialized, Role: FOLLOWER
I20260620 23:13:39.899617 27422 consensus_queue.cc:260] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6158b53b340ab8ebf10e6a39a80d7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38429 } } peers { permanent_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44977 } } peers { permanent_uuid: "596a7e2f307a4854b21a3ce8a4591e2f" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38079 } }
I20260620 23:13:39.899645 27424 raft_consensus.cc:348] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6158b53b340ab8ebf10e6a39a80d7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38429 } } peers { permanent_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44977 } } peers { permanent_uuid: "596a7e2f307a4854b21a3ce8a4591e2f" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38079 } }
I20260620 23:13:39.899648 27423 raft_consensus.cc:348] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6158b53b340ab8ebf10e6a39a80d7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38429 } } peers { permanent_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44977 } } peers { permanent_uuid: "596a7e2f307a4854b21a3ce8a4591e2f" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38079 } }
I20260620 23:13:39.899820 27423 raft_consensus.cc:374] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:39.899819 27424 raft_consensus.cc:374] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:39.899863 27423 raft_consensus.cc:729] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1a1a5ef31944f28cb03dd8ccd8da38, State: Initialized, Role: FOLLOWER
I20260620 23:13:39.899871 27424 raft_consensus.cc:729] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596a7e2f307a4854b21a3ce8a4591e2f, State: Initialized, Role: FOLLOWER
I20260620 23:13:39.899931 27422 ts_tablet_manager.cc:1428] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:13:39.899993 27424 consensus_queue.cc:260] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6158b53b340ab8ebf10e6a39a80d7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38429 } } peers { permanent_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44977 } } peers { permanent_uuid: "596a7e2f307a4854b21a3ce8a4591e2f" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38079 } }
I20260620 23:13:39.900060 27271 heartbeater.cc:503] Master 127.16.79.62:46155 was elected leader, sending a full tablet report...
I20260620 23:13:39.900272 27424 ts_tablet_manager.cc:1428] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:13:39.900301 27405 heartbeater.cc:503] Master 127.16.79.62:46155 was elected leader, sending a full tablet report...
I20260620 23:13:39.900327 27423 consensus_queue.cc:260] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6158b53b340ab8ebf10e6a39a80d7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38429 } } peers { permanent_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44977 } } peers { permanent_uuid: "596a7e2f307a4854b21a3ce8a4591e2f" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38079 } }
I20260620 23:13:39.900772 27137 heartbeater.cc:503] Master 127.16.79.62:46155 was elected leader, sending a full tablet report...
I20260620 23:13:39.900846 27423 ts_tablet_manager.cc:1428] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260620 23:13:39.937170 27272 tablet.cc:2369] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:13:40.026376 27138 tablet.cc:2369] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:13:40.108489 27406 tablet.cc:2369] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:13:40.120986 27430 raft_consensus.cc:482] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:40.121109 27430 raft_consensus.cc:504] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6158b53b340ab8ebf10e6a39a80d7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38429 } } peers { permanent_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44977 } } peers { permanent_uuid: "596a7e2f307a4854b21a3ce8a4591e2f" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38079 } }
I20260620 23:13:40.121524 27430 leader_election.cc:288] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c6158b53b340ab8ebf10e6a39a80d7 (127.16.79.2:38429), 596a7e2f307a4854b21a3ce8a4591e2f (127.16.79.3:38079)
I20260620 23:13:40.124938 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:40.121923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:50583 (local address 127.16.79.3:38079)
0620 23:13:40.121978 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:40.121981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:40.122176 (+ 195us) server_negotiation.cc:407] Connection header received
0620 23:13:40.122219 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:40.122220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:40.122234 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:40.122305 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:40.122562 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:40.123090 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:40.124202 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:40.124571 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:40.124625 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:40.124632 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:40.124820 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 23:13:40.124859 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:13:40.124958 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:40.121829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:38079 (local address 127.16.79.1:50583)
0620 23:13:40.122082 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:40.122094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:40.122169 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:40.122384 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:40.122386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:40.122389 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:40.122476 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:40.122478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:40.123158 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:40.123159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:40.124117 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:40.124119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:40.124644 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:40.124645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:40.124751 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:40.124754 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:13:40.124808 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:13:40.124853 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":123,"threads_started":1}
I20260620 23:13:40.125036 27419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:40.121936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:58795 (local address 127.16.79.2:38429)
0620 23:13:40.121978 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:40.121981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:40.122330 (+ 349us) server_negotiation.cc:407] Connection header received
0620 23:13:40.122358 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:40.122359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:40.122373 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:40.122451 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:40.122633 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:40.123160 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:40.124186 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:40.124651 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:40.124750 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:40.124760 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:40.124930 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 23:13:40.124967 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:13:40.125110 27431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:40.121829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:38429 (local address 127.16.79.1:58795)
0620 23:13:40.122181 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:40.122197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:40.122287 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:40.122460 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:40.122461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:40.122462 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:40.122559 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:40.122561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:40.123246 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:40.123247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:40.124081 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:40.124083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:40.124764 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:40.124765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:40.124850 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:40.124854 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:13:40.124907 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 23:13:40.124942 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":32,"thread_start_us":97,"threads_started":1}
I20260620 23:13:40.125414 27225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619a71c834d84e82b940f7facecefa2f" candidate_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6158b53b340ab8ebf10e6a39a80d7" is_pre_election: true
I20260620 23:13:40.125469 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619a71c834d84e82b940f7facecefa2f" candidate_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596a7e2f307a4854b21a3ce8a4591e2f" is_pre_election: true
I20260620 23:13:40.125573 27225 raft_consensus.cc:2454] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1a1a5ef31944f28cb03dd8ccd8da38 in term 0.
I20260620 23:13:40.125619 27359 raft_consensus.cc:2454] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1a1a5ef31944f28cb03dd8ccd8da38 in term 0.
I20260620 23:13:40.125779 27026 leader_election.cc:302] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6158b53b340ab8ebf10e6a39a80d7, 6b1a1a5ef31944f28cb03dd8ccd8da38; no voters:
I20260620 23:13:40.125950 27430 raft_consensus.cc:2781] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:13:40.126021 27430 raft_consensus.cc:482] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:40.126056 27430 raft_consensus.cc:3037] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:40.126739 27430 raft_consensus.cc:504] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6158b53b340ab8ebf10e6a39a80d7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38429 } } peers { permanent_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44977 } } peers { permanent_uuid: "596a7e2f307a4854b21a3ce8a4591e2f" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38079 } }
I20260620 23:13:40.126906 27430 leader_election.cc:288] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [CANDIDATE]: Term 1 election: Requested vote from peers 18c6158b53b340ab8ebf10e6a39a80d7 (127.16.79.2:38429), 596a7e2f307a4854b21a3ce8a4591e2f (127.16.79.3:38079)
I20260620 23:13:40.127053 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619a71c834d84e82b940f7facecefa2f" candidate_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596a7e2f307a4854b21a3ce8a4591e2f"
I20260620 23:13:40.127053 27225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619a71c834d84e82b940f7facecefa2f" candidate_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6158b53b340ab8ebf10e6a39a80d7"
I20260620 23:13:40.127147 27359 raft_consensus.cc:3037] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:40.127146 27225 raft_consensus.cc:3037] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:40.127767 27359 raft_consensus.cc:2454] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1a1a5ef31944f28cb03dd8ccd8da38 in term 1.
I20260620 23:13:40.127952 27225 raft_consensus.cc:2454] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1a1a5ef31944f28cb03dd8ccd8da38 in term 1.
I20260620 23:13:40.127975 27025 leader_election.cc:302] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596a7e2f307a4854b21a3ce8a4591e2f, 6b1a1a5ef31944f28cb03dd8ccd8da38; no voters:
I20260620 23:13:40.128157 27430 raft_consensus.cc:2781] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:40.128283 27430 raft_consensus.cc:686] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [term 1 LEADER]: Becoming Leader. State: Replica: 6b1a1a5ef31944f28cb03dd8ccd8da38, State: Running, Role: LEADER
I20260620 23:13:40.128377 27430 consensus_queue.cc:237] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6158b53b340ab8ebf10e6a39a80d7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38429 } } peers { permanent_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44977 } } peers { permanent_uuid: "596a7e2f307a4854b21a3ce8a4591e2f" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38079 } }
I20260620 23:13:40.129237 26946 catalog_manager.cc:5495] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b1a1a5ef31944f28cb03dd8ccd8da38 (127.16.79.1). New cstate: current_term: 1 leader_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6158b53b340ab8ebf10e6a39a80d7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b1a1a5ef31944f28cb03dd8ccd8da38" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596a7e2f307a4854b21a3ce8a4591e2f" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38079 } health_report { overall_health: UNKNOWN } } }
I20260620 23:13:40.177838 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:40.175364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.16.79.1:44977)
0620 23:13:40.175449 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:40.175451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:40.175470 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:13:40.175592 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:40.175593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:40.175612 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:40.175681 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:13:40.175858 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:40.176318 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:40.177142 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:40.177329 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:40.177392 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:40.177520 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:13:40.177610 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:13:40.177677 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:40.177712 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 23:13:40.177728 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:13:40.179692 27359 raft_consensus.cc:1261] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f [term 1 FOLLOWER]: Refusing update from remote peer 6b1a1a5ef31944f28cb03dd8ccd8da38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:40.179690 27225 raft_consensus.cc:1261] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7 [term 1 FOLLOWER]: Refusing update from remote peer 6b1a1a5ef31944f28cb03dd8ccd8da38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:40.180042 27433 consensus_queue.cc:1035] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596a7e2f307a4854b21a3ce8a4591e2f" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:40.180174 27430 consensus_queue.cc:1035] T 619a71c834d84e82b940f7facecefa2f P 6b1a1a5ef31944f28cb03dd8ccd8da38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c6158b53b340ab8ebf10e6a39a80d7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:40.182329 27436 mvcc.cc:205] Tried to move back new op lower bound from 7299060613853245440 to 7299060613646512128. Current Snapshot: MvccSnapshot[applied={T|T < 7299060613853245440}]
I20260620 23:13:40.184346 26945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:40.184433 26945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:40.185175 26945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0a85cfb65fc4b5e867d783d21124621: Sending DeleteTablet for 3 replicas of tablet 619a71c834d84e82b940f7facecefa2f
I20260620 23:13:40.185469 27205 tablet_service.cc:1514] Processing DeleteTablet for tablet 619a71c834d84e82b940f7facecefa2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:40 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:39074
I20260620 23:13:40.185477 27071 tablet_service.cc:1514] Processing DeleteTablet for tablet 619a71c834d84e82b940f7facecefa2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:40 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:41874
I20260620 23:13:40.185648 27339 tablet_service.cc:1514] Processing DeleteTablet for tablet 619a71c834d84e82b940f7facecefa2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:40 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:49996
I20260620 23:13:40.185783 27446 tablet_replica.cc:330] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7: stopping tablet replica
I20260620 23:13:40.185858 27448 tablet_replica.cc:330] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f: stopping tablet replica
I20260620 23:13:40.185953 27448 raft_consensus.cc:2229] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:40.185953 27446 raft_consensus.cc:2229] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:40.186048 27448 raft_consensus.cc:2258] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:40.186079 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 27008
I20260620 23:13:40.186188 27446 raft_consensus.cc:2258] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:40.186718 27446 ts_tablet_manager.cc:1905] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:40.188053 27448 ts_tablet_manager.cc:1905] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:40.188346 27446 ts_tablet_manager.cc:1918] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:40.188434 27446 log.cc:1196] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/619a71c834d84e82b940f7facecefa2f
I20260620 23:13:40.188683 27446 ts_tablet_manager.cc:1939] T 619a71c834d84e82b940f7facecefa2f P 18c6158b53b340ab8ebf10e6a39a80d7: Deleting consensus metadata
I20260620 23:13:40.189074 26932 catalog_manager.cc:4841] TS 18c6158b53b340ab8ebf10e6a39a80d7 (127.16.79.2:38429): tablet 619a71c834d84e82b940f7facecefa2f (table test-table [id=3b3a382713c845d59ceee45c8058ebe4]) successfully deleted
I20260620 23:13:40.189916 27448 ts_tablet_manager.cc:1918] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:40.189989 27448 log.cc:1196] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/619a71c834d84e82b940f7facecefa2f
I20260620 23:13:40.190233 27448 ts_tablet_manager.cc:1939] T 619a71c834d84e82b940f7facecefa2f P 596a7e2f307a4854b21a3ce8a4591e2f: Deleting consensus metadata
I20260620 23:13:40.190982 26931 catalog_manager.cc:4841] TS 596a7e2f307a4854b21a3ce8a4591e2f (127.16.79.3:38079): tablet 619a71c834d84e82b940f7facecefa2f (table test-table [id=3b3a382713c845d59ceee45c8058ebe4]) successfully deleted
W20260620 23:13:40.192292 26933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:44977 (error 108)
I20260620 23:13:40.192806 27417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:40.192649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:44977 (local address 127.0.0.1:41884)
0620 23:13:40.192737 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:40.192774 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:13:40.192821 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 27142
W20260620 23:13:40.196907 26933 catalog_manager.cc:4584] TS 6b1a1a5ef31944f28cb03dd8ccd8da38 (127.16.79.1:44977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 619a71c834d84e82b940f7facecefa2f: Network error: Client connection negotiation failed: client connection to 127.16.79.1:44977: connect: Connection refused (error 111)
I20260620 23:13:40.198781 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 27276
I20260620 23:13:40.204519 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 26914
2026-06-20T23:13:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2150 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:13:40 dist-test-slave-pswb krb5kdc[27452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:13:40 dist-test-slave-pswb krb5kdc[27452](info): set up 2 sockets
Jun 20 23:13:40 dist-test-slave-pswb krb5kdc[27452](info): commencing operation
krb5kdc: starting...
W20260620 23:13:41.253633 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:13:41 dist-test-slave-pswb krb5kdc[27452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:13:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.79.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.79.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/rangeradmin_127.16.79.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/rangeradmin_127.16.79.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.79.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.79.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/rangerlookup_127.16.79.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/rangerlookup_127.16.79.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.79.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.79.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/HTTP_127.16.79.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/HTTP_127.16.79.20@KRBTEST.COM.keytab.
I20260620 23:13:41.409359 16700 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmNabNW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/postgres -l logfile start
2026-06-20 23:13:43.956 UTC [27482] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:13:43.956 UTC [27482] LOG: listening on IPv4 address "127.16.79.20", port 33707
2026-06-20 23:13:43.960 UTC [27482] LOG: listening on Unix socket "/tmp/.s.PGSQL.33707"
2026-06-20 23:13:43.981 UTC [27485] LOG: database system was shut down at 2026-06-20 23:13:42 UTC
2026-06-20 23:13:43.984 UTC [27482] LOG: database system is ready to accept connections
I20260620 23:13:44.953718 16700 mini_postgres.cc:96] Postgres bound to 33707
2026-06-20 23:13:44.959 UTC [27493] FATAL: database "slave" does not exist
127.16.79.20:33707 - accepting connections
I20260620 23:13:44.959931 16700 mini_ranger.cc:161] Starting Ranger...
I20260620 23:13:44.973891 16700 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:13:45.477363 16700 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:13:45.477447 16700 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-admin
2026-06-20 23:13:45,746 [I] DB FLAVOR :POSTGRES
2026-06-20 23:13:45,746 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:13:45,746 [I] Checking connection..
2026-06-20 23:13:45,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:13:46,001 [I] Checking connection passed.
2026-06-20 23:13:46,001 [I] --------- Verifying version history table ---------
2026-06-20 23:13:46,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:13:46,247 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:13:46,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:13:46,483 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:13:46,483 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:13:46,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:13:46.704 UTC [27557] WARNING: there is no transaction in progress
2026-06-20 23:13:46,710 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:13:46,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:13:46,945 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:13:46,945 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:13:46,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:13:47,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:13:47,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pswb.c.gcp-upstream.internal','N') ;"
2026-06-20 23:13:47,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:13:47,880 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:13:47,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:13:48,135 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:13:48,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:13:48,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:13:48,629 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:13:48,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:13:49.569 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.585 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.601 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.617 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.632 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.649 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.703 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.711 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.726 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.737 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.749 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.764 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.774 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.780 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.787 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:49.794 UTC [27665] WARNING: there is no transaction in progress
2026-06-20 23:13:50,447 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:13:50,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:13:50,687 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:13:50,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:13:50,939 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:13:50,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:13:51,170 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:13:51,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:13:51,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:13:51,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pswb.c.gcp-upstream.internal';"
2026-06-20 23:13:51,887 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:13:51,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:13:52,148 [I] DB_PATCHES have already been applied
I20260620 23:13:52.154906 16700 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmNabNW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:13:52.154989 16700 mini_ranger.cc:193] Using host: 127.16.79.20
I20260620 23:13:52.158366 16700 mini_ranger.cc:227] Ranger admin URL: http://127.16.79.20:33973
Jun 20, 2026 11:13:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:13:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:13:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:13:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:13:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:13:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:13:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:13:53.252087 16700 mini_ranger.cc:160] Time spent starting Ranger: real 8.292s user 0.000s sys 0.009s
Jun 20, 2026 11:14:13 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:14:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 11:14:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:14:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:14:13 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:14:14 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:14:15.320528 16700 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.79.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.79.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/rangerkms_127.16.79.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/rangerkms_127.16.79.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.79.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.79.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.79.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/HTTP_127.16.79.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.79.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/HTTP_127.16.79.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:14:15 dist-test-slave-pswb krb5kdc[27452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997255, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:14:15.415078 16700 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:14:15.426671 16700 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:14:16.422322 16700 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:14:16.422426 16700 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-kms
I20260620 23:14:16.422446 16700 mini_ranger_kms.cc:227] Using postgres at 127.16.79.20:33707
2026-06-20 23:14:16,594 [I] DB FLAVOR :POSTGRES
2026-06-20 23:14:16,595 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:14:16,595 [I] Checking connection
2026-06-20 23:14:16,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:14:16,833 [I] connection success
2026-06-20 23:14:16,833 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:14:16,833 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:14:16,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:14:17,079 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:14:17,079 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:14:17,079 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:14:17,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:33707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:14:17,329 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:14:17.839161 16700 mini_ranger_kms.cc:310] Created kms service
I20260620 23:14:18.068264 16700 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:14:18.109140 16700 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:14:18 dist-test-slave-pswb krb5kdc[27452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997255, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:14:18.491326 16700 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:14:18.491432 16700 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:14:18.491456 16700 mini_ranger_kms.cc:240] Using host: 127.16.79.20
I20260620 23:14:18.494993 16700 mini_ranger_kms.cc:276] Ranger KMS PID: 27845
I20260620 23:14:18.495113 16700 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.79.20:57257
Jun 20, 2026 11:14:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:14:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:14:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:14:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:14:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:14:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 23:14:20.749433 16700 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.334s user 0.000s sys 0.155s
I20260620 23:14:20.749542 16700 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:14:20.749624 16700 mini_ranger_kms.cc:401] 127.16.79.20:57257/kms/v1/keys
Jun 20 23:14:20 dist-test-slave-pswb krb5kdc[27452](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997260, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.79.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:14:21 dist-test-slave-pswb krb5kdc[27452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997260, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.79.20@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
Jun 20 23:14:21 dist-test-slave-pswb krb5kdc[27452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997260, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.79.20@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
Jun 20 23:14:23 dist-test-slave-pswb krb5kdc[27452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:14:23.803813 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:34329
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:34329
--ranger_config_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.79.20:57257/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:23.887637 27896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:23.887864 27896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:23.887923 27896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:23.889045 27896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:14:23.889083 27896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:23.889097 27896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:23.889110 27896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:14:23.889122 27896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:14:23.890597 27896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.79.20:57257/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:34329
--ranger_config_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:34329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:14:23.890825 27896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:23.891047 27896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:23.895955 27902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:23.895918 27904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:23.895913 27901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:23.896214 27896 server_base.cc:1034] running on GCE node
I20260620 23:14:23.896448 27896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:23.896773 27896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:23.897944 27896 hybrid_clock.cc:648] HybridClock initialized: now 1781997263897923 us; error 31 us; skew 500 ppm
Jun 20 23:14:23 dist-test-slave-pswb krb5kdc[27452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997263, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:23.900678 27896 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:14:23.901450 27896 webserver.cc:460] Webserver started at http://127.16.79.62:39917/ using document root <none> and password file <none>
I20260620 23:14:23.901731 27896 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:23.901800 27896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:23.902122 27896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:14:23 dist-test-slave-pswb krb5kdc[27452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997263, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:14:23.950307 27896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "32b17e72743641da8a49f813fd27cbf5"
format_stamp: "Formatted at 2026-06-20 23:14:23 on dist-test-slave-pswb"
server_key: "cbab6a18c634545993b14c63b0a199e3"
server_key_iv: "310935efc538642817d174c4dc07b98e"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:23.950800 27896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "32b17e72743641da8a49f813fd27cbf5"
format_stamp: "Formatted at 2026-06-20 23:14:23 on dist-test-slave-pswb"
server_key: "cbab6a18c634545993b14c63b0a199e3"
server_key_iv: "310935efc538642817d174c4dc07b98e"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:23.952361 27896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:14:23.963457 27911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:23.963675 27896 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:14:23.963793 27896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "32b17e72743641da8a49f813fd27cbf5"
format_stamp: "Formatted at 2026-06-20 23:14:23 on dist-test-slave-pswb"
server_key: "cbab6a18c634545993b14c63b0a199e3"
server_key_iv: "310935efc538642817d174c4dc07b98e"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:23.963878 27896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:23.984593 27896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:23.986817 27896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:23.986968 27896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:23.991746 27896 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:34329
I20260620 23:14:23.991809 27963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:34329 every 8 connection(s)
I20260620 23:14:23.992215 27896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:14:23.993201 27964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:23.995694 27964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5: Bootstrap starting.
I20260620 23:14:23.996299 27964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:23.996603 27964 log.cc:824] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:23.997556 27964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5: No bootstrap required, opened a new log
I20260620 23:14:23.999765 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 27896
I20260620 23:14:23.999881 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:14:24.000082 16700 external_mini_cluster.cc:1428] Setting key e1814032ec1e7e73b99b66499a8bb3c9
I20260620 23:14:24.001572 27964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b17e72743641da8a49f813fd27cbf5" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34329 } }
I20260620 23:14:24.001938 27964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:24.002038 27964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b17e72743641da8a49f813fd27cbf5, State: Initialized, Role: FOLLOWER
I20260620 23:14:24.002241 27964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b17e72743641da8a49f813fd27cbf5" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34329 } }
I20260620 23:14:24.002346 27964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:14:24.002393 27964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:14:24.002455 27964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:24.003707 27964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b17e72743641da8a49f813fd27cbf5" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34329 } }
I20260620 23:14:24.003996 27964 leader_election.cc:302] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32b17e72743641da8a49f813fd27cbf5; no voters:
I20260620 23:14:24.004359 27964 leader_election.cc:288] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:14:24.004374 27969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:14:24.004678 27969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [term 1 LEADER]: Becoming Leader. State: Replica: 32b17e72743641da8a49f813fd27cbf5, State: Running, Role: LEADER
I20260620 23:14:24.004734 27964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:14:24.006155 27969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b17e72743641da8a49f813fd27cbf5" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34329 } }
I20260620 23:14:24.006546 27969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32b17e72743641da8a49f813fd27cbf5. Latest consensus state: current_term: 1 leader_uuid: "32b17e72743641da8a49f813fd27cbf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b17e72743641da8a49f813fd27cbf5" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34329 } } }
I20260620 23:14:24.006699 27969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:24.006781 27970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32b17e72743641da8a49f813fd27cbf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b17e72743641da8a49f813fd27cbf5" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34329 } } }
I20260620 23:14:24.006862 27970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:24.007213 27964 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 20 23:14:24 dist-test-slave-pswb krb5kdc[27452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:14:24.009877 27983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:14:24.011317 27983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:14:24.014231 27983 catalog_manager.cc:1347] Generated new cluster ID: 6af512e531484980b6433ebb01ac6ce6
I20260620 23:14:24.014339 27983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:14:24.015069 27968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:24.000595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.16.79.62:34329)
0620 23:14:24.000829 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:24.000839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:24.000876 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:14:24.001505 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:24.001519 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:24.001622 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:24.001836 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:24.002618 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:24.004152 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:24.005295 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:24.005894 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:24.005994 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:24.009239 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:24.009306 (+ 67us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:24.009306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:24.009402 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:24.012545 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:24.012999 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:24.013002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:24.013002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:24.013116 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:24.013330 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:24.013331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:24.013332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:24.013844 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:24.014428 (+ 584us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:24.014498 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 23:14:24.014792 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":110,"threads_started":1}
I20260620 23:14:24.041970 27983 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:14:24.042737 27983 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:14:24.047891 27983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5: Generated new TSK 0
I20260620 23:14:24.048103 27983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:14:24.783746 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.79.20:57257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34329
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:24.882579 28009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:24.882802 28009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:24.882846 28009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:24.884073 28009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:24.884125 28009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:24.884166 28009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:14:24.885789 28009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.79.20:57257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:14:24.886047 28009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:24.886294 28009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:24.891167 28017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:24.891145 28015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:24.891924 28009 server_base.cc:1034] running on GCE node
W20260620 23:14:24.892113 28014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:24.892321 28009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:24.892576 28009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:24.893748 28009 hybrid_clock.cc:648] HybridClock initialized: now 1781997264893735 us; error 27 us; skew 500 ppm
Jun 20 23:14:24 dist-test-slave-pswb krb5kdc[27452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997264, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:24.897141 28009 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:14:24.897874 28009 webserver.cc:460] Webserver started at http://127.16.79.1:43455/ using document root <none> and password file <none>
I20260620 23:14:24.898144 28009 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:24.898197 28009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:24.898336 28009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:14:24 dist-test-slave-pswb krb5kdc[27452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997264, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:14:24.912195 28009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "50584c805c3e434ab5f892880833ad86"
format_stamp: "Formatted at 2026-06-20 23:14:24 on dist-test-slave-pswb"
server_key: "b4cac194a000dd831cce7f129eb47af0"
server_key_iv: "e355452dd677138f0b83fc734e384557"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:24.912778 28009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "50584c805c3e434ab5f892880833ad86"
format_stamp: "Formatted at 2026-06-20 23:14:24 on dist-test-slave-pswb"
server_key: "b4cac194a000dd831cce7f129eb47af0"
server_key_iv: "e355452dd677138f0b83fc734e384557"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:24.914242 28009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:14:24.925971 28024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:24.928830 28009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 23:14:24.928938 28009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "50584c805c3e434ab5f892880833ad86"
format_stamp: "Formatted at 2026-06-20 23:14:24 on dist-test-slave-pswb"
server_key: "b4cac194a000dd831cce7f129eb47af0"
server_key_iv: "e355452dd677138f0b83fc734e384557"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:24.929023 28009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:24.944496 28009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:24.946734 28009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:24.946875 28009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:24.947082 28009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:24.947461 28009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:24.947538 28009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:24.947594 28009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:24.947638 28009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:24.954285 28009 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:44551
I20260620 23:14:24.954340 28139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:44551 every 8 connection(s)
I20260620 23:14:24.954669 28009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:14:24.956591 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 28009
I20260620 23:14:24.956701 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:14:24.956924 16700 external_mini_cluster.cc:1428] Setting key 9ee0ebbe8a2af7a936e45538b49e50da
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Jun 20 23:14:24 dist-test-slave-pswb krb5kdc[27452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997264, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:14:24.973397 28142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:24.955425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34329 (local address 127.16.79.1:54627)
0620 23:14:24.955588 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:24.955627 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:24.956089 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:24.964280 (+ 8191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:24.964285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:24.964461 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:24.964648 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:24.964652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:24.965584 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:24.965585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:24.966536 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:24.966539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:24.967066 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:24.967067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:24.967219 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:24.967815 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:24.967824 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:24.969489 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:24.971675 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:24.971681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:24.971684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:24.971997 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:24.972241 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:24.972242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:24.972242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:24.972365 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:24.972667 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:24.972673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:24.972891 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 23:14:24.973007 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 23:14:24.973156 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260620 23:14:24.974014 28143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:24.955459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:54627 (local address 127.16.79.62:34329)
0620 23:14:24.964029 (+ 8570us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:24.964031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:24.964040 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:14:24.964065 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:24.964066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:24.964079 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:24.964800 (+ 721us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:24.964891 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:24.965486 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:24.966625 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:24.966967 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:24.967038 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:24.969606 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:24.969607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:24.969608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:24.969615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:24.971574 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:24.972103 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:24.972104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:24.972104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:24.972171 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:24.972439 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:24.972440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:24.972441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:24.972579 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:24.972640 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:24.973767 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0620 23:14:24.973871 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8537,"thread_start_us":74,"threads_started":1}
I20260620 23:14:24.974519 28140 heartbeater.cc:347] Connected to a master server at 127.16.79.62:34329
I20260620 23:14:24.974629 28140 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:24.974821 28140 heartbeater.cc:511] Master 127.16.79.62:34329 requested a full tablet report, sending...
I20260620 23:14:24.975332 27922 ts_manager.cc:194] Registered new tserver with Master: 50584c805c3e434ab5f892880833ad86 (127.16.79.1:44551)
I20260620 23:14:24.976058 27922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:54627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:14:24.981805 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.79.20:57257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34329
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:25.099742 28146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:25.100028 28146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:25.100075 28146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:25.101965 28146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:25.102025 28146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:25.102067 28146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:14:25.104354 28146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.79.20:57257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:14:25.104607 28146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:25.104948 28146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:25.111143 28153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:25.111155 28154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:25.111155 28156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:25.111979 28146 server_base.cc:1034] running on GCE node
I20260620 23:14:25.112181 28146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:25.112438 28146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:25.113607 28146 hybrid_clock.cc:648] HybridClock initialized: now 1781997265113587 us; error 35 us; skew 500 ppm
Jun 20 23:14:25 dist-test-slave-pswb krb5kdc[27452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997265, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:25.116890 28146 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:14:25.117409 28146 webserver.cc:460] Webserver started at http://127.16.79.2:46315/ using document root <none> and password file <none>
I20260620 23:14:25.117686 28146 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:25.117754 28146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:25.117936 28146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:14:25 dist-test-slave-pswb krb5kdc[27452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997265, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
Jun 20 23:14:25 dist-test-slave-pswb krb5kdc[27452](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997265, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:25.139314 28146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "341c5c8309ab4ffa8e8f1a9577d28a05"
format_stamp: "Formatted at 2026-06-20 23:14:25 on dist-test-slave-pswb"
server_key: "257aa479122ba6e5f6903f88bb77f28e"
server_key_iv: "8e755f3718fb83cbf2955688933a88c9"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:25.139762 28146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "341c5c8309ab4ffa8e8f1a9577d28a05"
format_stamp: "Formatted at 2026-06-20 23:14:25 on dist-test-slave-pswb"
server_key: "257aa479122ba6e5f6903f88bb77f28e"
server_key_iv: "8e755f3718fb83cbf2955688933a88c9"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:25.142570 28146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:14:25.169546 28163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:25.169824 28146 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.001s
I20260620 23:14:25.169936 28146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "341c5c8309ab4ffa8e8f1a9577d28a05"
format_stamp: "Formatted at 2026-06-20 23:14:25 on dist-test-slave-pswb"
server_key: "257aa479122ba6e5f6903f88bb77f28e"
server_key_iv: "8e755f3718fb83cbf2955688933a88c9"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:25.170028 28146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:25.181521 28146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:25.183912 28146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:25.184103 28146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:25.184376 28146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:25.184800 28146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:25.184890 28146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:25.184969 28146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:25.185025 28146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:25.191375 28146 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:38093
I20260620 23:14:25.191783 28146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:14:25.196142 28276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:38093 every 8 connection(s)
I20260620 23:14:25.197824 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 28146
I20260620 23:14:25.197930 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:14:25.198127 16700 external_mini_cluster.cc:1428] Setting key 0f508e5338018ccfdcba15a2915dd8a4
Jun 20 23:14:25 dist-test-slave-pswb krb5kdc[27452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997265, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:14:25.208755 28143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.197289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:58507 (local address 127.16.79.62:34329)
0620 23:14:25.197331 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:25.197333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:25.197343 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:14:25.198094 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:25.198095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:25.198109 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:25.198175 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:25.198836 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.199574 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.201100 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.201410 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.201450 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.205433 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:25.205434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:25.205435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:25.205441 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:25.207165 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:25.207637 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:25.207639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:25.207639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:25.207696 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:25.207989 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:25.207990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:25.207991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:25.208122 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:25.208165 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:25.208538 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 23:14:25.208625 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
I20260620 23:14:25.209271 28279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.197024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34329 (local address 127.16.79.2:58507)
0620 23:14:25.197216 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:25.197262 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:25.197917 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:25.198249 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:25.198253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:25.198476 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:25.198738 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.198744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.199680 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.199680 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.201005 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.201008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.201533 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.201534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.201667 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.203777 (+ 2110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:25.203787 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:25.205329 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:25.207260 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:25.207266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:25.207268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:25.207544 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:25.207769 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:25.207770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:25.207771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:25.207893 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:25.208214 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:25.208219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:25.208379 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 23:14:25.208883 (+ 504us) client_negotiation.cc:240] Negotiation successful
0620 23:14:25.209053 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260620 23:14:25.209837 28277 heartbeater.cc:347] Connected to a master server at 127.16.79.62:34329
I20260620 23:14:25.209926 28277 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:25.210117 28277 heartbeater.cc:511] Master 127.16.79.62:34329 requested a full tablet report, sending...
I20260620 23:14:25.210454 27922 ts_manager.cc:194] Registered new tserver with Master: 341c5c8309ab4ffa8e8f1a9577d28a05 (127.16.79.2:38093)
I20260620 23:14:25.210920 27922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:58507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:14:25.236008 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.79.20:57257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34329
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:25.366475 28283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:25.366797 28283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:25.366879 28283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:25.368563 28283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:25.368620 28283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:25.368659 28283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:14:25.370762 28283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.79.20:57257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:14:25.371193 28283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:25.371505 28283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:25.381111 28288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:25.382154 28289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:25.382313 28283 server_base.cc:1034] running on GCE node
W20260620 23:14:25.382373 28291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:25.382665 28283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:25.382915 28283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:25.384089 28283 hybrid_clock.cc:648] HybridClock initialized: now 1781997265384070 us; error 36 us; skew 500 ppm
Jun 20 23:14:25 dist-test-slave-pswb krb5kdc[27452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997265, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:25.387301 28283 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:14:25.387706 28283 webserver.cc:460] Webserver started at http://127.16.79.3:45383/ using document root <none> and password file <none>
I20260620 23:14:25.387951 28283 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:25.388020 28283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:25.388156 28283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:14:25 dist-test-slave-pswb krb5kdc[27452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997265, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:14:25.402432 28283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "53bdabe72eb1420f9fe9b89f2c230751"
format_stamp: "Formatted at 2026-06-20 23:14:25 on dist-test-slave-pswb"
server_key: "5f667b1b7a5578bcb09bd68afdb1eb51"
server_key_iv: "660b965c52c062bf9e3e2aa81e14fd6c"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:25.402915 28283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "53bdabe72eb1420f9fe9b89f2c230751"
format_stamp: "Formatted at 2026-06-20 23:14:25 on dist-test-slave-pswb"
server_key: "5f667b1b7a5578bcb09bd68afdb1eb51"
server_key_iv: "660b965c52c062bf9e3e2aa81e14fd6c"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:25.404958 28283 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:14:25.423914 28298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:25.424155 28283 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260620 23:14:25.424249 28283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "53bdabe72eb1420f9fe9b89f2c230751"
format_stamp: "Formatted at 2026-06-20 23:14:25 on dist-test-slave-pswb"
server_key: "5f667b1b7a5578bcb09bd68afdb1eb51"
server_key_iv: "660b965c52c062bf9e3e2aa81e14fd6c"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:25.424336 28283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:25.457645 28283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:25.459990 28283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:25.460170 28283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:25.460423 28283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:25.460875 28283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:25.460968 28283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:25.461050 28283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:25.461112 28283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:25.467530 28283 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:36541
I20260620 23:14:25.468047 28283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:14:25.472064 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 28283
I20260620 23:14:25.472164 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:14:25.472345 16700 external_mini_cluster.cc:1428] Setting key 754c5131507f52969ab1fca0d79bc17b
I20260620 23:14:25.481112 28411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:36541 every 8 connection(s)
Jun 20 23:14:25 dist-test-slave-pswb krb5kdc[27452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997265, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:14:25.496594 28143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.481992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:55497 (local address 127.16.79.62:34329)
0620 23:14:25.482048 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:25.482050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:25.485536 (+ 3486us) server_negotiation.cc:407] Connection header received
0620 23:14:25.486079 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:25.486080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:25.486092 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:25.486124 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:25.486949 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.487716 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.489353 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.489644 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.489685 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.493096 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:25.493097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:25.493097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:25.493104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:25.494733 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:25.495534 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:25.495535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:25.495536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:25.495596 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:25.495867 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:25.495867 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:25.495868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:25.495990 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:25.496061 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:25.496417 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 23:14:25.496498 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:14:25.497123 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.481728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34329 (local address 127.16.79.3:55497)
0620 23:14:25.485377 (+ 3649us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:25.485424 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:25.485941 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:25.486315 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:25.486319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:25.486571 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:25.486841 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.486846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.487835 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.487836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.489251 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.489254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.489770 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.489771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.489904 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.491368 (+ 1464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:25.491378 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:25.492972 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:25.494812 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:25.494818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:25.494821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:25.495131 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:25.495669 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:25.495670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:25.495670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:25.495792 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:25.496098 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:25.496103 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:25.496262 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 23:14:25.496744 (+ 482us) client_negotiation.cc:240] Negotiation successful
0620 23:14:25.496891 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3603,"thread_start_us":84,"threads_started":1}
I20260620 23:14:25.497524 28412 heartbeater.cc:347] Connected to a master server at 127.16.79.62:34329
I20260620 23:14:25.497622 28412 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:25.497808 28412 heartbeater.cc:511] Master 127.16.79.62:34329 requested a full tablet report, sending...
I20260620 23:14:25.498143 27922 ts_manager.cc:194] Registered new tserver with Master: 53bdabe72eb1420f9fe9b89f2c230751 (127.16.79.3:36541)
I20260620 23:14:25.498633 27922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:55497
I20260620 23:14:25.505688 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:14:25.520058 28143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.509004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.16.79.62:34329)
0620 23:14:25.509043 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:25.509045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:25.512413 (+ 3368us) server_negotiation.cc:407] Connection header received
0620 23:14:25.512514 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:25.512515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:25.512526 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:25.512555 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:25.513333 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.514099 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.515171 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.515472 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.515514 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.517210 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:25.517211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:25.517211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:25.517218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:25.518862 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:25.519243 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:25.519244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:25.519245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:25.519296 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:25.519500 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:25.519500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:25.519501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:25.519615 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:25.519687 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:25.519896 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 23:14:25.519970 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:14:25.521921 27922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:14:25.524134 27922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:14:25.548146 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.536217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:38093 (local address 127.0.0.1:39456)
0620 23:14:25.538037 (+ 1820us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:25.538045 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:25.538093 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:25.538740 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:25.538741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:25.538742 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:25.538772 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.538773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.542662 (+ 3889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.542663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.543867 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.543869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.547854 (+ 3985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.547855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.547929 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.547942 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:14:25.548006 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:14:25.548065 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1805,"thread_start_us":70,"threads_started":1}
I20260620 23:14:25.550172 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.535995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:44551 (local address 127.0.0.1:48628)
0620 23:14:25.537745 (+ 1750us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:25.537784 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:25.537852 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:25.538591 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:25.538594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:25.538596 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:25.538669 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.538671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.540685 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.540686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.548000 (+ 7314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.548002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.549935 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.549936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.550011 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.550014 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:14:25.550065 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 23:14:25.550103 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1722,"thread_start_us":69,"threads_started":1}
I20260620 23:14:25.550418 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.536976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:36541 (local address 127.0.0.1:49940)
0620 23:14:25.543983 (+ 7007us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:25.543992 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:25.544034 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:25.544561 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:25.544562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:25.544564 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:25.544603 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.544604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.545872 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.545873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.546964 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.546967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.550223 (+ 3256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.550224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.550281 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.550284 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:14:25.550330 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 23:14:25.550362 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6991,"thread_start_us":68,"threads_started":1}
I20260620 23:14:25.551194 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.537199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.16.79.3:36541)
0620 23:14:25.537990 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:25.537994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:25.544090 (+ 6096us) server_negotiation.cc:407] Connection header received
0620 23:14:25.544133 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:25.544136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:25.544151 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:25.544218 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:25.544675 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.545778 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.547081 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.549193 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.549264 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.549327 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:25.551071 (+ 1744us) server_negotiation.cc:299] Negotiation successful
0620 23:14:25.551110 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":75,"threads_started":1}
I20260620 23:14:25.551985 28346 tablet_service.cc:1467] Processing CreateTablet for tablet e3b713437e8c499f9ed6298113278e4c (DEFAULT_TABLE table=test-table [id=f68027bdac7a4f09931368b7447c16c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:25.552354 28346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b713437e8c499f9ed6298113278e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:25.556190 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.536502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.16.79.1:44551)
0620 23:14:25.538382 (+ 1880us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:25.538386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:25.538394 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:14:25.538424 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:25.538425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:25.538454 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:25.538525 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:25.539505 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.540587 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.548453 (+ 7866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.549783 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.549855 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.549901 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:25.556073 (+ 6172us) server_negotiation.cc:299] Negotiation successful
0620 23:14:25.556119 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1852,"thread_start_us":53,"threads_started":1}
I20260620 23:14:25.557016 28074 tablet_service.cc:1467] Processing CreateTablet for tablet e3b713437e8c499f9ed6298113278e4c (DEFAULT_TABLE table=test-table [id=f68027bdac7a4f09931368b7447c16c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:25.557305 28074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b713437e8c499f9ed6298113278e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:25.560755 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.536634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.16.79.2:38093)
0620 23:14:25.537675 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:25.537680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:25.538148 (+ 468us) server_negotiation.cc:407] Connection header received
0620 23:14:25.538190 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:25.538193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:25.538210 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:25.538277 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:25.538841 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.542565 (+ 3724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.546634 (+ 4069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.547381 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.547450 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.547523 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:25.560605 (+ 13082us) server_negotiation.cc:299] Negotiation successful
0620 23:14:25.560653 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1004,"thread_start_us":109,"threads_started":1}
I20260620 23:14:25.561681 28211 tablet_service.cc:1467] Processing CreateTablet for tablet e3b713437e8c499f9ed6298113278e4c (DEFAULT_TABLE table=test-table [id=f68027bdac7a4f09931368b7447c16c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:25.561971 28211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b713437e8c499f9ed6298113278e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:25.563225 28431 tablet_bootstrap.cc:492] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751: Bootstrap starting.
I20260620 23:14:25.564213 28431 tablet_bootstrap.cc:654] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:25.564363 28432 tablet_bootstrap.cc:492] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86: Bootstrap starting.
I20260620 23:14:25.565034 28432 tablet_bootstrap.cc:654] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:25.565402 28431 log.cc:824] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:25.565598 28433 tablet_bootstrap.cc:492] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05: Bootstrap starting.
I20260620 23:14:25.566118 28432 log.cc:824] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:25.566689 28433 tablet_bootstrap.cc:654] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:25.566972 28433 log.cc:824] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:25.567005 28431 tablet_bootstrap.cc:492] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751: No bootstrap required, opened a new log
I20260620 23:14:25.567061 28431 ts_tablet_manager.cc:1397] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:14:25.568133 28433 tablet_bootstrap.cc:492] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05: No bootstrap required, opened a new log
I20260620 23:14:25.568210 28433 ts_tablet_manager.cc:1397] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:14:25.569527 28431 raft_consensus.cc:348] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bdabe72eb1420f9fe9b89f2c230751" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36541 } } peers { permanent_uuid: "50584c805c3e434ab5f892880833ad86" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44551 } } peers { permanent_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38093 } }
I20260620 23:14:25.569772 28431 raft_consensus.cc:374] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:25.569856 28431 raft_consensus.cc:729] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53bdabe72eb1420f9fe9b89f2c230751, State: Initialized, Role: FOLLOWER
I20260620 23:14:25.570048 28431 consensus_queue.cc:260] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bdabe72eb1420f9fe9b89f2c230751" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36541 } } peers { permanent_uuid: "50584c805c3e434ab5f892880833ad86" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44551 } } peers { permanent_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38093 } }
I20260620 23:14:25.570194 28433 raft_consensus.cc:348] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bdabe72eb1420f9fe9b89f2c230751" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36541 } } peers { permanent_uuid: "50584c805c3e434ab5f892880833ad86" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44551 } } peers { permanent_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38093 } }
I20260620 23:14:25.570343 28433 raft_consensus.cc:374] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:25.570389 28433 raft_consensus.cc:729] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341c5c8309ab4ffa8e8f1a9577d28a05, State: Initialized, Role: FOLLOWER
I20260620 23:14:25.570526 28431 ts_tablet_manager.cc:1428] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:14:25.570492 28433 consensus_queue.cc:260] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bdabe72eb1420f9fe9b89f2c230751" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36541 } } peers { permanent_uuid: "50584c805c3e434ab5f892880833ad86" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44551 } } peers { permanent_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38093 } }
I20260620 23:14:25.570691 28412 heartbeater.cc:503] Master 127.16.79.62:34329 was elected leader, sending a full tablet report...
I20260620 23:14:25.570708 28433 ts_tablet_manager.cc:1428] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:14:25.570822 28277 heartbeater.cc:503] Master 127.16.79.62:34329 was elected leader, sending a full tablet report...
I20260620 23:14:25.571837 28432 tablet_bootstrap.cc:492] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86: No bootstrap required, opened a new log
I20260620 23:14:25.571900 28432 ts_tablet_manager.cc:1397] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260620 23:14:25.574178 28432 raft_consensus.cc:348] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bdabe72eb1420f9fe9b89f2c230751" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36541 } } peers { permanent_uuid: "50584c805c3e434ab5f892880833ad86" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44551 } } peers { permanent_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38093 } }
I20260620 23:14:25.574314 28432 raft_consensus.cc:374] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:25.574353 28432 raft_consensus.cc:729] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50584c805c3e434ab5f892880833ad86, State: Initialized, Role: FOLLOWER
I20260620 23:14:25.574474 28432 consensus_queue.cc:260] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bdabe72eb1420f9fe9b89f2c230751" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36541 } } peers { permanent_uuid: "50584c805c3e434ab5f892880833ad86" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44551 } } peers { permanent_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38093 } }
I20260620 23:14:25.574736 28432 ts_tablet_manager.cc:1428] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:14:25.574770 28140 heartbeater.cc:503] Master 127.16.79.62:34329 was elected leader, sending a full tablet report...
Jun 20 23:14:25 dist-test-slave-pswb krb5kdc[27452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997265, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:14:25.690675 28437 raft_consensus.cc:482] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:14:25.690864 28437 raft_consensus.cc:504] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bdabe72eb1420f9fe9b89f2c230751" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36541 } } peers { permanent_uuid: "50584c805c3e434ab5f892880833ad86" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44551 } } peers { permanent_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38093 } }
I20260620 23:14:25.691402 28437 leader_election.cc:288] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50584c805c3e434ab5f892880833ad86 (127.16.79.1:44551), 341c5c8309ab4ffa8e8f1a9577d28a05 (127.16.79.2:38093)
I20260620 23:14:25.696233 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.691772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:44551 (local address 127.16.79.3:37433)
0620 23:14:25.691818 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:25.691834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:25.691890 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:25.692165 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:25.692166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:25.692169 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:25.692264 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.692266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.693719 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.693722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.695062 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.695064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.695979 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.695980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.696057 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.696061 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:14:25.696106 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 23:14:25.696144 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 23:14:25.696839 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.691916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:37433 (local address 127.16.79.1:44551)
0620 23:14:25.691965 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:25.691967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:25.691976 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:14:25.692008 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:25.692008 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:25.692042 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:25.692094 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:25.692908 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.693590 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.695293 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.695861 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.695924 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.695936 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:25.696717 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 23:14:25.696758 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:14:25.696974 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.692803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:34443 (local address 127.16.79.2:38093)
0620 23:14:25.692854 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:25.692856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:25.693715 (+ 859us) server_negotiation.cc:407] Connection header received
0620 23:14:25.693826 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:25.693827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:25.693836 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:25.693865 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:25.694076 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.694757 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.696067 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.696571 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.696620 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.696630 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:25.696870 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 23:14:25.696910 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:14:25.697125 28094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b713437e8c499f9ed6298113278e4c" candidate_uuid: "53bdabe72eb1420f9fe9b89f2c230751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50584c805c3e434ab5f892880833ad86" is_pre_election: true
W20260620 23:14:25.697198 28278 tablet.cc:2369] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:14:25.697301 28094 raft_consensus.cc:2454] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53bdabe72eb1420f9fe9b89f2c230751 in term 0.
I20260620 23:14:25.697533 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.692434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:38093 (local address 127.16.79.3:34443)
0620 23:14:25.693640 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:25.693656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:25.693747 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:25.693929 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:25.693929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:25.693931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:25.694003 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.694005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.694865 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.694866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.695980 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:25.695982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.696720 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.696722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:25.696787 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.696790 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:14:25.697041 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 23:14:25.697417 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"thread_start_us":138,"threads_started":1}
I20260620 23:14:25.697533 28301 leader_election.cc:302] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50584c805c3e434ab5f892880833ad86, 53bdabe72eb1420f9fe9b89f2c230751; no voters:
I20260620 23:14:25.698000 28437 raft_consensus.cc:2781] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:14:25.697979 28231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b713437e8c499f9ed6298113278e4c" candidate_uuid: "53bdabe72eb1420f9fe9b89f2c230751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05" is_pre_election: true
I20260620 23:14:25.698055 28437 raft_consensus.cc:482] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:14:25.698081 28437 raft_consensus.cc:3037] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:25.698112 28231 raft_consensus.cc:2454] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53bdabe72eb1420f9fe9b89f2c230751 in term 0.
I20260620 23:14:25.698807 28437 raft_consensus.cc:504] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bdabe72eb1420f9fe9b89f2c230751" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36541 } } peers { permanent_uuid: "50584c805c3e434ab5f892880833ad86" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44551 } } peers { permanent_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38093 } }
I20260620 23:14:25.698994 28437 leader_election.cc:288] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [CANDIDATE]: Term 1 election: Requested vote from peers 50584c805c3e434ab5f892880833ad86 (127.16.79.1:44551), 341c5c8309ab4ffa8e8f1a9577d28a05 (127.16.79.2:38093)
I20260620 23:14:25.699134 28094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b713437e8c499f9ed6298113278e4c" candidate_uuid: "53bdabe72eb1420f9fe9b89f2c230751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50584c805c3e434ab5f892880833ad86"
I20260620 23:14:25.699208 28094 raft_consensus.cc:3037] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:25.699265 28231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b713437e8c499f9ed6298113278e4c" candidate_uuid: "53bdabe72eb1420f9fe9b89f2c230751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05"
I20260620 23:14:25.699326 28231 raft_consensus.cc:3037] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:25.700099 28094 raft_consensus.cc:2454] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53bdabe72eb1420f9fe9b89f2c230751 in term 1.
I20260620 23:14:25.700232 28231 raft_consensus.cc:2454] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53bdabe72eb1420f9fe9b89f2c230751 in term 1.
I20260620 23:14:25.700263 28301 leader_election.cc:302] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50584c805c3e434ab5f892880833ad86, 53bdabe72eb1420f9fe9b89f2c230751; no voters:
I20260620 23:14:25.700444 28437 raft_consensus.cc:2781] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:14:25.700609 28437 raft_consensus.cc:686] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 1 LEADER]: Becoming Leader. State: Replica: 53bdabe72eb1420f9fe9b89f2c230751, State: Running, Role: LEADER
I20260620 23:14:25.700722 28437 consensus_queue.cc:237] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bdabe72eb1420f9fe9b89f2c230751" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36541 } } peers { permanent_uuid: "50584c805c3e434ab5f892880833ad86" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44551 } } peers { permanent_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38093 } }
I20260620 23:14:25.701488 27924 catalog_manager.cc:5495] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53bdabe72eb1420f9fe9b89f2c230751 (127.16.79.3). New cstate: current_term: 1 leader_uuid: "53bdabe72eb1420f9fe9b89f2c230751" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bdabe72eb1420f9fe9b89f2c230751" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50584c805c3e434ab5f892880833ad86" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38093 } health_report { overall_health: UNKNOWN } } }
W20260620 23:14:25.705583 28141 tablet.cc:2369] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:14:25.724411 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.721461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.16.79.3:36541)
0620 23:14:25.721508 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:25.721510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:25.721520 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:14:25.721576 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:25.721577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:25.721597 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:25.721663 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:14:25.721827 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.722540 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.723535 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:25.723825 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:25.723892 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:25.724079 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:14:25.724192 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:14:25.724272 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:25.724322 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 23:14:25.724343 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:14:25.726140 28231 raft_consensus.cc:1261] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05 [term 1 FOLLOWER]: Refusing update from remote peer 53bdabe72eb1420f9fe9b89f2c230751: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:25.726228 28094 raft_consensus.cc:1261] T e3b713437e8c499f9ed6298113278e4c P 50584c805c3e434ab5f892880833ad86 [term 1 FOLLOWER]: Refusing update from remote peer 53bdabe72eb1420f9fe9b89f2c230751: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:25.726549 28437 consensus_queue.cc:1035] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341c5c8309ab4ffa8e8f1a9577d28a05" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:25.726670 28441 consensus_queue.cc:1035] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50584c805c3e434ab5f892880833ad86" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:25.730379 28444 mvcc.cc:205] Tried to move back new op lower bound from 7299060800411893760 to 7299060800310702080. Current Snapshot: MvccSnapshot[applied={T|T < 7299060800411893760}]
I20260620 23:14:25.731016 28445 mvcc.cc:205] Tried to move back new op lower bound from 7299060800411893760 to 7299060800310702080. Current Snapshot: MvccSnapshot[applied={T|T < 7299060800411893760}]
I20260620 23:14:25.731103 28447 mvcc.cc:205] Tried to move back new op lower bound from 7299060800411893760 to 7299060800310702080. Current Snapshot: MvccSnapshot[applied={T|T < 7299060800411893760 or (T in {7299060800411893760})}]
I20260620 23:14:25.733817 27922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:14:25.733901 27922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:14:25.734622 27922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32b17e72743641da8a49f813fd27cbf5: Sending DeleteTablet for 3 replicas of tablet e3b713437e8c499f9ed6298113278e4c
I20260620 23:14:25.734979 28074 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b713437e8c499f9ed6298113278e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:25 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:48628
I20260620 23:14:25.734995 28211 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b713437e8c499f9ed6298113278e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:25 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:39456
I20260620 23:14:25.735188 28454 tablet_replica.cc:330] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05: stopping tablet replica
I20260620 23:14:25.735270 28454 raft_consensus.cc:2229] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:14:25.735311 28346 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b713437e8c499f9ed6298113278e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:25 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:49940
I20260620 23:14:25.735365 28454 raft_consensus.cc:2258] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:25.735512 28455 tablet_replica.cc:330] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751: stopping tablet replica
I20260620 23:14:25.735582 28455 raft_consensus.cc:2229] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:14:25.735673 28455 raft_consensus.cc:2258] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:25.736029 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 28009
I20260620 23:14:25.736055 28454 ts_tablet_manager.cc:1905] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:25.738692 28455 ts_tablet_manager.cc:1905] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:25.739305 28454 ts_tablet_manager.cc:1918] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:25.739387 28454 log.cc:1196] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/e3b713437e8c499f9ed6298113278e4c
I20260620 23:14:25.739657 28454 ts_tablet_manager.cc:1939] T e3b713437e8c499f9ed6298113278e4c P 341c5c8309ab4ffa8e8f1a9577d28a05: Deleting consensus metadata
I20260620 23:14:25.740033 27915 catalog_manager.cc:4841] TS 341c5c8309ab4ffa8e8f1a9577d28a05 (127.16.79.2:38093): tablet e3b713437e8c499f9ed6298113278e4c (table test-table [id=f68027bdac7a4f09931368b7447c16c3]) successfully deleted
I20260620 23:14:25.740154 28455 ts_tablet_manager.cc:1918] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:25.740224 28455 log.cc:1196] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/e3b713437e8c499f9ed6298113278e4c
I20260620 23:14:25.740489 28455 ts_tablet_manager.cc:1939] T e3b713437e8c499f9ed6298113278e4c P 53bdabe72eb1420f9fe9b89f2c230751: Deleting consensus metadata
I20260620 23:14:25.740904 27913 catalog_manager.cc:4841] TS 53bdabe72eb1420f9fe9b89f2c230751 (127.16.79.3:36541): tablet e3b713437e8c499f9ed6298113278e4c (table test-table [id=f68027bdac7a4f09931368b7447c16c3]) successfully deleted
W20260620 23:14:25.743073 27914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:44551 (error 108)
I20260620 23:14:25.743633 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 28146
I20260620 23:14:25.744951 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.743688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:44551 (local address 127.0.0.1:48644)
0620 23:14:25.744878 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:25.744919 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1165}
W20260620 23:14:25.745796 27914 catalog_manager.cc:4584] TS 50584c805c3e434ab5f892880833ad86 (127.16.79.1:44551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3b713437e8c499f9ed6298113278e4c: Network error: Client connection negotiation failed: client connection to 127.16.79.1:44551: connect: Connection refused (error 111)
I20260620 23:14:25.748598 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 28283
I20260620 23:14:25.754130 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 27896
I20260620 23:14:25.759316 16700 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:14:26.104249 16700 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:14:26.104348 16700 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:14:26.448906 16700 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:14:26.449 UTC [27482] LOG: received smart shutdown request
2026-06-20 23:14:26.453 UTC [27482] LOG: background worker "logical replication launcher" (PID 27491) exited with exit code 1
2026-06-20 23:14:26.453 UTC [27486] LOG: shutting down
2026-06-20 23:14:26.638 UTC [27482] LOG: database system is shut down
2026-06-20T23:14:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46543 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:14:26 dist-test-slave-pswb krb5kdc[28465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:14:26 dist-test-slave-pswb krb5kdc[28465](info): set up 2 sockets
Jun 20 23:14:26 dist-test-slave-pswb krb5kdc[28465](info): commencing operation
krb5kdc: starting...
W20260620 23:14:27.806057 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:14:27 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:14:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.79.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.79.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/rangeradmin_127.16.79.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/rangeradmin_127.16.79.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.79.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.79.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/rangerlookup_127.16.79.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/rangerlookup_127.16.79.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.79.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.79.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/HTTP_127.16.79.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/HTTP_127.16.79.20@KRBTEST.COM.keytab.
I20260620 23:14:27.975397 16700 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmNabNW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/postgres -l logfile start
2026-06-20 23:14:30.765 UTC [28495] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:14:30.765 UTC [28495] LOG: listening on IPv4 address "127.16.79.20", port 48205
2026-06-20 23:14:30.772 UTC [28495] LOG: listening on Unix socket "/tmp/.s.PGSQL.48205"
2026-06-20 23:14:30.810 UTC [28498] LOG: database system was shut down at 2026-06-20 23:14:29 UTC
2026-06-20 23:14:30.815 UTC [28495] LOG: database system is ready to accept connections
I20260620 23:14:31.784526 16700 mini_postgres.cc:96] Postgres bound to 48205
2026-06-20 23:14:31.790 UTC [28506] FATAL: database "slave" does not exist
127.16.79.20:48205 - accepting connections
I20260620 23:14:31.791013 16700 mini_ranger.cc:161] Starting Ranger...
I20260620 23:14:31.807119 16700 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:14:32.367151 16700 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:14:32.367257 16700 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-admin
2026-06-20 23:14:32,652 [I] DB FLAVOR :POSTGRES
2026-06-20 23:14:32,652 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:14:32,652 [I] Checking connection..
2026-06-20 23:14:32,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:14:32,887 [I] Checking connection passed.
2026-06-20 23:14:32,887 [I] --------- Verifying version history table ---------
2026-06-20 23:14:32,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:14:33,119 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:14:33,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:14:33,363 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:14:33,363 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:14:33,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:14:33.596 UTC [28570] WARNING: there is no transaction in progress
2026-06-20 23:14:33,604 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:14:33,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:14:33,862 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:14:33,862 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:14:33,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:14:34,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:14:34,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pswb.c.gcp-upstream.internal','N') ;"
2026-06-20 23:14:34,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:14:34,837 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:14:34,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:14:35,081 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:14:35,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:14:35,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:14:35,571 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:14:35,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:14:36.581 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.598 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.618 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.635 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.652 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.671 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.726 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.732 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.749 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.759 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.770 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.784 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.793 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.799 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.807 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:36.813 UTC [28678] WARNING: there is no transaction in progress
2026-06-20 23:14:37,491 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:14:37,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:14:37,723 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:14:37,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:14:37,980 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:14:37,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:14:38,250 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:14:38,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:14:38,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:14:38,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pswb.c.gcp-upstream.internal';"
2026-06-20 23:14:38,998 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:14:38,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:14:39,223 [I] DB_PATCHES have already been applied
I20260620 23:14:39.229993 16700 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmNabNW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:14:39.230051 16700 mini_ranger.cc:193] Using host: 127.16.79.20
I20260620 23:14:39.233029 16700 mini_ranger.cc:227] Ranger admin URL: http://127.16.79.20:51515
Jun 20, 2026 11:14:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:14:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:14:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:14:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:14:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:14:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:14:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:14:40.335019 16700 mini_ranger.cc:160] Time spent starting Ranger: real 8.544s user 0.000s sys 0.008s
Jun 20, 2026 11:14:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:14:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 11:14:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:14:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:14:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:15:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:15:00.843853 16700 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.79.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.79.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/rangerkms_127.16.79.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.79.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/rangerkms_127.16.79.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.79.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.79.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.79.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/HTTP_127.16.79.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.79.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/HTTP_127.16.79.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:15:00 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997300, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:15:00.943162 16700 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:15:00.953194 16700 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:15:02.070952 16700 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:15:02.071038 16700 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-kms
I20260620 23:15:02.071058 16700 mini_ranger_kms.cc:227] Using postgres at 127.16.79.20:48205
2026-06-20 23:15:02,257 [I] DB FLAVOR :POSTGRES
2026-06-20 23:15:02,257 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:15:02,257 [I] Checking connection
2026-06-20 23:15:02,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:15:02,496 [I] connection success
2026-06-20 23:15:02,496 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:15:02,497 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:15:02,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:15:02,738 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:15:02,738 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:15:02,738 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:15:02,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.79.20:48205/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:15:02,998 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:15:03.493659 16700 mini_ranger_kms.cc:310] Created kms service
I20260620 23:15:03.670912 16700 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:15:03.713198 16700 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:15:03 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997300, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:15:04.094161 16700 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:15:04.094271 16700 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmNabNW/build/release/bin/postgresql.jar:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmNabNW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:15:04.094291 16700 mini_ranger_kms.cc:240] Using host: 127.16.79.20
I20260620 23:15:04.100981 16700 mini_ranger_kms.cc:276] Ranger KMS PID: 28858
I20260620 23:15:04.101074 16700 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.79.20:33769
Jun 20, 2026 11:15:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:15:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:15:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:15:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:15:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:15:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 23:15:06.397092 16700 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.454s user 0.001s sys 0.167s
I20260620 23:15:06.397181 16700 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:15:06.397228 16700 mini_ranger_kms.cc:401] 127.16.79.20:33769/kms/v1/keys
Jun 20 23:15:06 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997306, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.79.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:15:06 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997306, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.79.20@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
Jun 20 23:15:07 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997306, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.79.20@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
Jun 20 23:15:09 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:15:09.431396 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36753
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:36753
--ranger_config_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.79.20:33769/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:09.514853 28909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:09.515075 28909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:09.515107 28909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:09.516414 28909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:09.516453 28909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:09.516467 28909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:09.516480 28909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:09.516491 28909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:09.518127 28909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.79.20:33769/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:36753
--ranger_config_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:09.518368 28909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:09.518610 28909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:15:09.524921 28909 server_base.cc:1034] running on GCE node
W20260620 23:15:09.524868 28915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:09.524875 28914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:09.524952 28917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:09.525311 28909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:09.525604 28909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:09.526824 28909 hybrid_clock.cc:648] HybridClock initialized: now 1781997309526807 us; error 31 us; skew 500 ppm
Jun 20 23:15:09 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997309, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:09.533952 28909 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:15:09.534526 28909 webserver.cc:460] Webserver started at http://127.16.79.62:36085/ using document root <none> and password file <none>
I20260620 23:15:09.534783 28909 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:09.534847 28909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:09.535013 28909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:15:09 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997309, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:15:09.584183 28909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "d9cc4074d37846048e24fe3692004be4"
format_stamp: "Formatted at 2026-06-20 23:15:09 on dist-test-slave-pswb"
server_key: "523134e3665bcc7ef56d23866191317c"
server_key_iv: "57b0875afe8deadfe4bdab153bbc939d"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:09.584599 28909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "d9cc4074d37846048e24fe3692004be4"
format_stamp: "Formatted at 2026-06-20 23:15:09 on dist-test-slave-pswb"
server_key: "523134e3665bcc7ef56d23866191317c"
server_key_iv: "57b0875afe8deadfe4bdab153bbc939d"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:09.586050 28909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:09.596740 28924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:09.596927 28909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:15:09.597033 28909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "d9cc4074d37846048e24fe3692004be4"
format_stamp: "Formatted at 2026-06-20 23:15:09 on dist-test-slave-pswb"
server_key: "523134e3665bcc7ef56d23866191317c"
server_key_iv: "57b0875afe8deadfe4bdab153bbc939d"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:09.597102 28909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:09.607806 28909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:09.610085 28909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:09.610241 28909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:09.615449 28909 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:36753
I20260620 23:15:09.615540 28976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:36753 every 8 connection(s)
I20260620 23:15:09.615866 28909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:09.616592 28977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:09.617007 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 28909
I20260620 23:15:09.617106 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:15:09.617290 16700 external_mini_cluster.cc:1428] Setting key 781b1ec94c71e654df4709ac4bbb1b56
I20260620 23:15:09.619547 28977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4: Bootstrap starting.
I20260620 23:15:09.620260 28977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:09.620613 28977 log.cc:824] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:09.621717 28977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4: No bootstrap required, opened a new log
I20260620 23:15:09.624015 28977 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc4074d37846048e24fe3692004be4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36753 } }
I20260620 23:15:09.624192 28977 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:09.624226 28977 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9cc4074d37846048e24fe3692004be4, State: Initialized, Role: FOLLOWER
I20260620 23:15:09.624362 28977 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc4074d37846048e24fe3692004be4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36753 } }
I20260620 23:15:09.624432 28977 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:09.624459 28977 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:09.624501 28977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:09.625181 28977 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc4074d37846048e24fe3692004be4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36753 } }
I20260620 23:15:09.625288 28977 leader_election.cc:302] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9cc4074d37846048e24fe3692004be4; no voters:
I20260620 23:15:09.625443 28977 leader_election.cc:288] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:15:09.625559 28982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:09.625677 28977 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:09.625809 28982 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 1 LEADER]: Becoming Leader. State: Replica: d9cc4074d37846048e24fe3692004be4, State: Running, Role: LEADER
I20260620 23:15:09.625973 28982 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc4074d37846048e24fe3692004be4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36753 } }
I20260620 23:15:09.626341 28982 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9cc4074d37846048e24fe3692004be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc4074d37846048e24fe3692004be4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36753 } } }
I20260620 23:15:09.626482 28982 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:09.626667 28982 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9cc4074d37846048e24fe3692004be4. Latest consensus state: current_term: 1 leader_uuid: "d9cc4074d37846048e24fe3692004be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc4074d37846048e24fe3692004be4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36753 } } }
I20260620 23:15:09.626868 28982 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:09.627357 28977 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 20 23:15:09 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:09.631379 28992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:09.632230 28992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:09.634984 28992 catalog_manager.cc:1347] Generated new cluster ID: 52e21e68be1f4561bc5720f6d2f9eba5
I20260620 23:15:09.635033 28992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:09.636386 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:09.617779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39562 (local address 127.16.79.62:36753)
0620 23:15:09.621422 (+ 3643us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:09.621462 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:09.621586 (+ 124us) server_negotiation.cc:407] Connection header received
0620 23:15:09.622419 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:09.622430 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:09.622495 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:09.622652 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:09.623284 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:09.624676 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:09.625585 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:09.626204 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:09.626313 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:09.630453 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:09.630515 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:09.630516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:09.630642 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:09.633942 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:09.634370 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:09.634373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:09.634373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:09.634483 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:09.634687 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:09.634688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:09.634689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:09.635095 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:09.635248 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:09.635714 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 23:15:09.636029 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3551,"thread_start_us":77,"threads_started":1}
I20260620 23:15:09.665477 28992 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:15:09.666394 28992 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:09.694527 28992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4: Generated new TSK 0
I20260620 23:15:09.694857 28992 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:15:10.391184 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.79.20:33769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:36753
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:10.476233 29022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:10.476459 29022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:10.476500 29022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:10.477666 29022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:10.477732 29022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:10.477777 29022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:15:10.479189 29022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.79.20:33769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:36753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:10.479436 29022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:10.479665 29022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:10.484468 29030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:10.484593 29028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:10.484835 29027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:10.484987 29022 server_base.cc:1034] running on GCE node
I20260620 23:15:10.485260 29022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:10.485519 29022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:10.486688 29022 hybrid_clock.cc:648] HybridClock initialized: now 1781997310486670 us; error 35 us; skew 500 ppm
Jun 20 23:15:10 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997310, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:10.490464 29022 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:15:10.491108 29022 webserver.cc:460] Webserver started at http://127.16.79.1:37511/ using document root <none> and password file <none>
I20260620 23:15:10.491431 29022 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:10.491554 29022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:10.491693 29022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:15:10 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997310, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:15:10.505427 29022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "8fbfefac38b04f72ae981093e9b98f3d"
format_stamp: "Formatted at 2026-06-20 23:15:10 on dist-test-slave-pswb"
server_key: "690b6877a0750a8e136edb35d5a7235c"
server_key_iv: "3cafbb5bb8a6fbe48278778ce39fc675"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:10.505963 29022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "8fbfefac38b04f72ae981093e9b98f3d"
format_stamp: "Formatted at 2026-06-20 23:15:10 on dist-test-slave-pswb"
server_key: "690b6877a0750a8e136edb35d5a7235c"
server_key_iv: "3cafbb5bb8a6fbe48278778ce39fc675"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:10.509523 29022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:15:10.523023 29037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:10.523236 29022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:10.523357 29022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "8fbfefac38b04f72ae981093e9b98f3d"
format_stamp: "Formatted at 2026-06-20 23:15:10 on dist-test-slave-pswb"
server_key: "690b6877a0750a8e136edb35d5a7235c"
server_key_iv: "3cafbb5bb8a6fbe48278778ce39fc675"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:10.523484 29022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:10.557654 29022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:10.560632 29022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:10.560896 29022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:10.561264 29022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:10.561741 29022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:10.561828 29022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:10.561944 29022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:10.562027 29022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:10.571578 29022 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:44099
I20260620 23:15:10.571854 29152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:44099 every 8 connection(s)
I20260620 23:15:10.572139 29022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:15:10 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997310, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:10.579802 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 29022
I20260620 23:15:10.579898 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:15:10.580099 16700 external_mini_cluster.cc:1428] Setting key 4321425d8a5f20a43944f11fff8d0976
I20260620 23:15:10.582820 29156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:10.573188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:54085 (local address 127.16.79.62:36753)
0620 23:15:10.574284 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:10.574287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:10.574296 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:15:10.574323 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:10.574323 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:10.574335 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:10.574410 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:10.575045 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:10.575857 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:10.577014 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:10.577215 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:10.577326 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:10.579345 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:10.579347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:10.579347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:10.579354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:10.581279 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:10.581762 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:10.581763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:10.581764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:10.581825 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:10.582099 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:10.582100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:10.582100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:10.582237 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:10.582300 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:10.582644 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 23:15:10.582723 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":88,"threads_started":1}
I20260620 23:15:10.583015 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:10.573149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:36753 (local address 127.16.79.1:54085)
0620 23:15:10.573367 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:10.573418 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:10.574099 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:10.574505 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:10.574509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:10.574733 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:10.574940 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:10.574944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:10.575996 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:10.575997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:10.576917 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:10.576919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:10.577347 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:10.577348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:10.577468 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:10.577993 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:10.578001 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:10.579251 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:10.581370 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:10.581376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:10.581378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:10.581683 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:10.581888 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:10.581889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:10.581889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:10.582032 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:10.582328 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:10.582335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:10.582505 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 23:15:10.582621 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 23:15:10.582761 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":108,"threads_started":1}
I20260620 23:15:10.583681 29153 heartbeater.cc:347] Connected to a master server at 127.16.79.62:36753
I20260620 23:15:10.583796 29153 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:10.583976 29153 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
I20260620 23:15:10.584476 28941 ts_manager.cc:194] Registered new tserver with Master: 8fbfefac38b04f72ae981093e9b98f3d (127.16.79.1:44099)
I20260620 23:15:10.585459 28941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:54085
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:15:10.611189 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.79.20:33769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:36753
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:10.724624 29159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:10.725088 29159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:10.725199 29159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:10.727952 29159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:10.728058 29159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:10.728120 29159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:15:10.731189 29159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.79.20:33769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:36753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:10.731479 29159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:10.731705 29159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 23:15:10 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997310, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 23:15:10.736892 29169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:10.736925 29166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:10.737151 29159 server_base.cc:1034] running on GCE node
W20260620 23:15:10.737354 29167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:10.737586 29159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:10.737852 29159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:10.739019 29159 hybrid_clock.cc:648] HybridClock initialized: now 1781997310739005 us; error 25 us; skew 500 ppm
Jun 20 23:15:10 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997310, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:10.741583 29159 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:15:10.742065 29159 webserver.cc:460] Webserver started at http://127.16.79.2:40461/ using document root <none> and password file <none>
I20260620 23:15:10.742305 29159 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:10.742377 29159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:10.742516 29159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:15:10 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997310, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:15:10.755863 29159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "141418c867ce406994e8cae68484bdbf"
format_stamp: "Formatted at 2026-06-20 23:15:10 on dist-test-slave-pswb"
server_key: "e7b93d7ebca7515b66300d1b55a75175"
server_key_iv: "8272759780d03064c2bd5230a2456baa"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:10.756286 29159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "141418c867ce406994e8cae68484bdbf"
format_stamp: "Formatted at 2026-06-20 23:15:10 on dist-test-slave-pswb"
server_key: "e7b93d7ebca7515b66300d1b55a75175"
server_key_iv: "8272759780d03064c2bd5230a2456baa"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:10.770519 29159 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.003s sys 0.000s
I20260620 23:15:10.788825 29176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:10.789006 29159 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260620 23:15:10.789093 29159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "141418c867ce406994e8cae68484bdbf"
format_stamp: "Formatted at 2026-06-20 23:15:10 on dist-test-slave-pswb"
server_key: "e7b93d7ebca7515b66300d1b55a75175"
server_key_iv: "8272759780d03064c2bd5230a2456baa"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:10.789186 29159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:10.812785 29159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:10.817224 29159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:10.817389 29159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:10.817631 29159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:10.818001 29159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:10.818048 29159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:10.818078 29159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:10.818110 29159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:10.824258 29159 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:40663
I20260620 23:15:10.824671 29159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:15:10.826963 29290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:40663 every 8 connection(s)
I20260620 23:15:10.828040 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 29159
I20260620 23:15:10.828155 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:15:10.828419 16700 external_mini_cluster.cc:1428] Setting key cd931754968d7b714c1a27317f8d7b5f
Jun 20 23:15:10 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997310, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:10.838647 29156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:10.828466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:42639 (local address 127.16.79.62:36753)
0620 23:15:10.828519 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:10.828521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:10.828576 (+ 55us) server_negotiation.cc:407] Connection header received
0620 23:15:10.829276 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:10.829277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:10.829289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:10.829319 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:10.830173 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:10.830947 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:10.832440 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:10.832782 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:10.832824 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:10.835394 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:10.835395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:10.835396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:10.835403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:10.837121 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:10.837589 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:10.837590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:10.837590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:10.837647 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:10.837910 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:10.837911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:10.837911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:10.838037 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:10.838077 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:10.838466 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 23:15:10.838552 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:15:10.839365 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:10.828179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:36753 (local address 127.16.79.2:42639)
0620 23:15:10.828389 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:10.828437 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:10.829153 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:10.829534 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:10.829538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:10.829781 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:10.830051 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:10.830057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:10.831064 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:10.831065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:10.832341 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:10.832344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:10.832923 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:10.832924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:10.833054 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:10.833629 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:10.833639 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:10.835287 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:10.837217 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:10.837222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:10.837225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:10.837501 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:10.837718 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:10.837719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:10.837719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:10.837837 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:10.838138 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:10.838144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:10.838312 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 23:15:10.838976 (+ 664us) client_negotiation.cc:240] Negotiation successful
0620 23:15:10.839143 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
I20260620 23:15:10.840210 29291 heartbeater.cc:347] Connected to a master server at 127.16.79.62:36753
I20260620 23:15:10.840298 29291 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:10.840473 29291 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
I20260620 23:15:10.840859 28941 ts_manager.cc:194] Registered new tserver with Master: 141418c867ce406994e8cae68484bdbf (127.16.79.2:40663)
I20260620 23:15:10.841344 28941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:42639
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:15:10.872654 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.79.20:33769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:36753
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:10.995015 29296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:10.995291 29296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:10.995332 29296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:10.997095 29296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:10.997148 29296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:10.997191 29296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:15:10.999285 29296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.79.20:33769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:36753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:10.999524 29296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:10.999805 29296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:11.006260 29302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:11.007093 29304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:11.007295 29301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:11.007385 29296 server_base.cc:1034] running on GCE node
I20260620 23:15:11.007586 29296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:11.007890 29296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:11.009074 29296 hybrid_clock.cc:648] HybridClock initialized: now 1781997311009044 us; error 44 us; skew 500 ppm
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:11.011966 29296 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:15:11.012413 29296 webserver.cc:460] Webserver started at http://127.16.79.3:33181/ using document root <none> and password file <none>
I20260620 23:15:11.012653 29296 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:11.012742 29296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:11.012882 29296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:15:11.026047 29296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "caa707d1452f48f1919335976a3916f7"
format_stamp: "Formatted at 2026-06-20 23:15:11 on dist-test-slave-pswb"
server_key: "256db59eeccf51b4682094d169c8359a"
server_key_iv: "4ad2a74ee051833c70d76ff8f63871af"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:11.026569 29296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "caa707d1452f48f1919335976a3916f7"
format_stamp: "Formatted at 2026-06-20 23:15:11 on dist-test-slave-pswb"
server_key: "256db59eeccf51b4682094d169c8359a"
server_key_iv: "4ad2a74ee051833c70d76ff8f63871af"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:11.030439 29296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 23:15:11.041640 29311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.041860 29296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:11.041985 29296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "caa707d1452f48f1919335976a3916f7"
format_stamp: "Formatted at 2026-06-20 23:15:11 on dist-test-slave-pswb"
server_key: "256db59eeccf51b4682094d169c8359a"
server_key_iv: "4ad2a74ee051833c70d76ff8f63871af"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:11.042120 29296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:11.055030 29296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:11.057288 29296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:11.057468 29296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:11.057755 29296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:11.058137 29296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:11.058228 29296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.058296 29296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:11.058354 29296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.067024 29296 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:44353
I20260620 23:15:11.067421 29296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:15:11.068084 29424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:44353 every 8 connection(s)
I20260620 23:15:11.069253 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 29296
I20260620 23:15:11.069365 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:15:11.069556 16700 external_mini_cluster.cc:1428] Setting key 0f479fb4c6e57b9e420abefb43e21fb0
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:11.080279 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:11.068341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:36753 (local address 127.16.79.3:44969)
0620 23:15:11.068520 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:11.068565 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:11.070124 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:11.070548 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:11.070552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:11.070777 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:11.071033 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:11.071038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.072047 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.072048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:11.073294 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:11.073297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.073853 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.073854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:11.074000 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:11.074576 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:11.074586 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:11.076949 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:11.078772 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:11.078777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:11.078779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:11.078986 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:11.079349 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:11.079351 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:11.079351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:11.079476 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:11.079808 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:11.079812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:11.079935 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 23:15:11.080013 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 23:15:11.080123 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260620 23:15:11.080277 29156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:11.068359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:44969 (local address 127.16.79.62:36753)
0620 23:15:11.068402 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:11.068404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:11.068652 (+ 248us) server_negotiation.cc:407] Connection header received
0620 23:15:11.070230 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:11.070231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:11.070253 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:11.070283 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:11.071154 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.071929 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.073401 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.073701 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.073749 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:11.077052 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:11.077053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:11.077054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:11.077061 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:11.078684 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:11.079105 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:11.079106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:11.079106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:11.079180 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:11.079580 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:11.079582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:11.079582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:11.079732 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:11.079821 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:11.080089 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 23:15:11.080183 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:15:11.080892 29425 heartbeater.cc:347] Connected to a master server at 127.16.79.62:36753
I20260620 23:15:11.080984 29425 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:11.081178 29425 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
I20260620 23:15:11.081549 28941 ts_manager.cc:194] Registered new tserver with Master: caa707d1452f48f1919335976a3916f7 (127.16.79.3:44353)
I20260620 23:15:11.081929 28941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:44969
I20260620 23:15:11.092074 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:15:11.092782 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 29022
I20260620 23:15:11.098315 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 29159
I20260620 23:15:11.103305 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 29296
I20260620 23:15:11.108889 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 28909
I20260620 23:15:11.113862 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36753
--webserver_interface=127.16.79.62
--webserver_port=36085
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:36753
--ranger_config_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.79.20:33769/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.79.20:33769/kms with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:11.193416 29428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:11.193598 29428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:11.193619 29428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:11.194703 29428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:11.194744 29428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:11.194758 29428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:11.194770 29428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:11.194782 29428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:11.196286 29428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.79.20:33769/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:36753
--ranger_config_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=36085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:11.196509 29428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:11.196754 29428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:11.201746 29436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:11.201758 29433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:11.202006 29434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:11.202425 29428 server_base.cc:1034] running on GCE node
I20260620 23:15:11.202818 29428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:11.203079 29428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:11.204277 29428 hybrid_clock.cc:648] HybridClock initialized: now 1781997311204256 us; error 46 us; skew 500 ppm
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:11.206748 29428 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:15:11.207226 29428 webserver.cc:460] Webserver started at http://127.16.79.62:36085/ using document root <none> and password file <none>
I20260620 23:15:11.207441 29428 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:11.207513 29428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:15:11.257084 29428 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:15:11.258009 29444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.258219 29428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:11.258356 29428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "d9cc4074d37846048e24fe3692004be4"
format_stamp: "Formatted at 2026-06-20 23:15:09 on dist-test-slave-pswb"
server_key: "523134e3665bcc7ef56d23866191317c"
server_key_iv: "57b0875afe8deadfe4bdab153bbc939d"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:11.258853 29428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:11.274789 29428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:11.276600 29428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:11.276808 29428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:11.281323 29428 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:36753
I20260620 23:15:11.281347 29496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:36753 every 8 connection(s)
I20260620 23:15:11.281713 29428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:11.282900 29497 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:15:11.283524 29497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4: Bootstrap starting.
I20260620 23:15:11.290032 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 29428
I20260620 23:15:11.290329 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.79.20:33769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:44099
--local_ip_for_outbound_sockets=127.16.79.1
--tserver_master_addrs=127.16.79.62:36753
--webserver_port=37511
--webserver_interface=127.16.79.1
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.79.20:33769/kms with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:11.291375 29497 log.cc:824] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:11.293007 29497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:15:11.293272 29497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4: Bootstrap complete.
I20260620 23:15:11.295547 29497 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc4074d37846048e24fe3692004be4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36753 } }
I20260620 23:15:11.295799 29497 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9cc4074d37846048e24fe3692004be4, State: Initialized, Role: FOLLOWER
I20260620 23:15:11.295970 29497 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc4074d37846048e24fe3692004be4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36753 } }
I20260620 23:15:11.296044 29497 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:11.296095 29497 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:11.296142 29497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:15:11.297147 29497 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc4074d37846048e24fe3692004be4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36753 } }
I20260620 23:15:11.297262 29497 leader_election.cc:302] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9cc4074d37846048e24fe3692004be4; no voters:
I20260620 23:15:11.297446 29497 leader_election.cc:288] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:15:11.297477 29501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:15:11.297603 29501 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [term 2 LEADER]: Becoming Leader. State: Replica: d9cc4074d37846048e24fe3692004be4, State: Running, Role: LEADER
I20260620 23:15:11.297703 29497 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:11.297690 29501 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc4074d37846048e24fe3692004be4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36753 } }
I20260620 23:15:11.297945 29501 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9cc4074d37846048e24fe3692004be4. Latest consensus state: current_term: 2 leader_uuid: "d9cc4074d37846048e24fe3692004be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc4074d37846048e24fe3692004be4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36753 } } }
I20260620 23:15:11.298018 29501 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:11.297950 29502 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d9cc4074d37846048e24fe3692004be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9cc4074d37846048e24fe3692004be4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36753 } } }
I20260620 23:15:11.298229 29502 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:11.298364 29505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:11.298861 29505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:11.299006 29505 catalog_manager.cc:1259] Loaded cluster ID: 52e21e68be1f4561bc5720f6d2f9eba5
I20260620 23:15:11.299052 29505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:11.299227 29497 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:15:11.302614 29505 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:11.303167 29505 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4: Loaded TSK: 0
I20260620 23:15:11.303494 29505 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:15:11.409551 29499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:11.409901 29499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:11.409981 29499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:11.411659 29499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:11.411759 29499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:11.411841 29499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:15:11.414042 29499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.79.20:33769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:44099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=37511
--tserver_master_addrs=127.16.79.62:36753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:11.414598 29499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:11.414922 29499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:11.421144 29537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:11.421247 29535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:11.421492 29499 server_base.cc:1034] running on GCE node
W20260620 23:15:11.421479 29534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:11.421851 29499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:11.422124 29499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:11.423235 29499 hybrid_clock.cc:648] HybridClock initialized: now 1781997311423207 us; error 43 us; skew 500 ppm
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:11.426514 29499 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:15:11.426964 29499 webserver.cc:460] Webserver started at http://127.16.79.1:37511/ using document root <none> and password file <none>
I20260620 23:15:11.427253 29499 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:11.427325 29499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:15:11.447899 29499 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:11.448760 29545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.448971 29499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:11.449072 29499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "8fbfefac38b04f72ae981093e9b98f3d"
format_stamp: "Formatted at 2026-06-20 23:15:10 on dist-test-slave-pswb"
server_key: "690b6877a0750a8e136edb35d5a7235c"
server_key_iv: "3cafbb5bb8a6fbe48278778ce39fc675"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:11.449513 29499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:11.461344 29499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:11.463661 29499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:11.463819 29499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:11.464048 29499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:11.464408 29499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:11.464461 29499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.464489 29499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:11.464507 29499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.472034 29499 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:44099
I20260620 23:15:11.472160 29658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:44099 every 8 connection(s)
I20260620 23:15:11.472450 29499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:15:11.478394 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 29499
I20260620 23:15:11.478693 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.79.20:33769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:40663
--local_ip_for_outbound_sockets=127.16.79.2
--tserver_master_addrs=127.16.79.62:36753
--webserver_port=40461
--webserver_interface=127.16.79.2
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.79.20:33769/kms with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:11.485447 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:11.473219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:36753 (local address 127.16.79.1:51895)
0620 23:15:11.473388 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:11.473422 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:11.474002 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:11.474505 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:11.474509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:11.474702 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:11.474875 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:11.474879 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.476064 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.476065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:11.476943 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:11.476945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.477570 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.477571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:11.477694 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:11.478200 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:11.478208 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:11.480910 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:11.483418 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:11.483425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:11.483427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:11.483738 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:11.484041 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:11.484041 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:11.484042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:11.484191 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:11.484720 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:11.484727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:11.484899 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 23:15:11.484993 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:15:11.485170 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260620 23:15:11.486773 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:11.473495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:51895 (local address 127.16.79.62:36753)
0620 23:15:11.473722 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:11.473730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:11.473767 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:15:11.474237 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:11.474244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:11.474297 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:11.474430 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:11.474986 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.475960 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.477056 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.477453 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.477579 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:11.481036 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:11.481040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:11.481040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:11.481052 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:11.483311 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:11.483843 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:11.483845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:11.483846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:11.483955 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:11.484280 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:11.484280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:11.484281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:11.484593 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:11.486041 (+ 1448us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:11.486197 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 23:15:11.486494 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":97,"threads_started":1}
I20260620 23:15:11.487525 29659 heartbeater.cc:347] Connected to a master server at 127.16.79.62:36753
I20260620 23:15:11.487617 29659 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:11.487800 29659 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
W20260620 23:15:11.488763 29659 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:36753 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:15:11.488842 29659 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:11.488919 29659 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
I20260620 23:15:11.489315 29659 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:11.489394 29659 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
W20260620 23:15:11.489745 29659 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:15:11.582289 29663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:11.582502 29663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:11.582540 29663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:11.583750 29663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:11.583807 29663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:11.583887 29663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:15:11.585551 29663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.79.20:33769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:40663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=40461
--tserver_master_addrs=127.16.79.62:36753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:11.585801 29663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:11.586015 29663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:11.590975 29668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:11.590956 29671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:11.590943 29669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:11.591413 29663 server_base.cc:1034] running on GCE node
I20260620 23:15:11.591596 29663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:11.591832 29663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:11.592994 29663 hybrid_clock.cc:648] HybridClock initialized: now 1781997311592975 us; error 33 us; skew 500 ppm
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:11.596043 29663 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:15:11.596525 29663 webserver.cc:460] Webserver started at http://127.16.79.2:40461/ using document root <none> and password file <none>
I20260620 23:15:11.596779 29663 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:11.596845 29663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:15:11.616461 29663 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:15:11.617297 29679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.617451 29663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:15:11.617530 29663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "141418c867ce406994e8cae68484bdbf"
format_stamp: "Formatted at 2026-06-20 23:15:10 on dist-test-slave-pswb"
server_key: "e7b93d7ebca7515b66300d1b55a75175"
server_key_iv: "8272759780d03064c2bd5230a2456baa"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:11.617983 29663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:11.655648 29663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:11.658416 29663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:11.658586 29663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:11.658854 29663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:11.659231 29663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:11.659271 29663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.659296 29663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:11.659309 29663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.667788 29663 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:40663
I20260620 23:15:11.667869 29792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:40663 every 8 connection(s)
I20260620 23:15:11.668221 29663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:11.675467 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 29663
I20260620 23:15:11.675833 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.79.20:33769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:44353
--local_ip_for_outbound_sockets=127.16.79.3
--tserver_master_addrs=127.16.79.62:36753
--webserver_port=33181
--webserver_interface=127.16.79.3
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.79.20:33769/kms with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:11.678452 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:11.668992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:49517 (local address 127.16.79.62:36753)
0620 23:15:11.669085 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:11.669087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:11.669340 (+ 253us) server_negotiation.cc:407] Connection header received
0620 23:15:11.669885 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:11.669887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:11.669899 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:11.669960 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:11.670537 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.671135 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.672228 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.672445 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.672530 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:11.675257 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:11.675259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:11.675259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:11.675267 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:11.676891 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:11.677283 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:11.677284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:11.677284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:11.677343 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:11.677628 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:11.677629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:11.677629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:11.677781 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:11.677850 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:11.678238 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 23:15:11.678330 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:15:11.678982 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:11.668955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:36753 (local address 127.16.79.2:49517)
0620 23:15:11.669201 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:11.669240 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:11.669780 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:11.670044 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:11.670048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:11.670215 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:11.670440 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:11.670445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.671227 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.671228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:11.672122 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:11.672124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.672551 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.672552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:11.672664 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:11.673214 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:11.673222 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:11.675160 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:11.676982 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:11.676988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:11.676991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:11.677206 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:11.677419 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:11.677420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:11.677420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:11.677560 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:11.677884 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:11.677891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:11.678067 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 23:15:11.678594 (+ 527us) client_negotiation.cc:240] Negotiation successful
0620 23:15:11.678746 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
I20260620 23:15:11.679518 29793 heartbeater.cc:347] Connected to a master server at 127.16.79.62:36753
I20260620 23:15:11.679617 29793 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:11.679805 29793 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
W20260620 23:15:11.680341 29793 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:36753 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:15:11.680415 29793 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:11.680487 29793 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
I20260620 23:15:11.680787 29793 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:11.680866 29793 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
W20260620 23:15:11.681128 29793 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:15:11.773221 29796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:11.773576 29796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:11.773658 29796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:11.775077 29796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:11.775142 29796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:11.775185 29796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:15:11.776598 29796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.79.20:33769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:44353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=33181
--tserver_master_addrs=127.16.79.62:36753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:11.776868 29796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:11.777117 29796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:11.782788 29804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:11.783238 29802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:11.783581 29796 server_base.cc:1034] running on GCE node
W20260620 23:15:11.785688 29801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:11.785964 29796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:11.786216 29796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:11.793648 29796 hybrid_clock.cc:648] HybridClock initialized: now 1781997311793619 us; error 36 us; skew 500 ppm
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:11.796267 29796 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:15:11.796831 29796 webserver.cc:460] Webserver started at http://127.16.79.3:33181/ using document root <none> and password file <none>
I20260620 23:15:11.797070 29796 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:11.797154 29796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:15:11.811203 29796 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:15:11.811879 29813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.812049 29796 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:11.812143 29796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "caa707d1452f48f1919335976a3916f7"
format_stamp: "Formatted at 2026-06-20 23:15:11 on dist-test-slave-pswb"
server_key: "256db59eeccf51b4682094d169c8359a"
server_key_iv: "4ad2a74ee051833c70d76ff8f63871af"
server_key_version: "kuduclusterkey@0"
I20260620 23:15:11.812508 29796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:11.824627 29796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:11.827098 29796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:11.827250 29796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:11.827478 29796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:11.827870 29796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:11.827920 29796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.827957 29796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:11.827978 29796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:11.835201 29796 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:44353
I20260620 23:15:11.835356 29926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:44353 every 8 connection(s)
I20260620 23:15:11.835696 29796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:15:11.845587 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 29796
Jun 20 23:15:11 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:11.853411 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:11.836755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:52999 (local address 127.16.79.62:36753)
0620 23:15:11.836804 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:11.836807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:11.840971 (+ 4164us) server_negotiation.cc:407] Connection header received
0620 23:15:11.841641 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:11.841643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:11.841656 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:11.841732 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:11.842520 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.843397 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.845291 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.845685 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.845787 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:11.849774 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:11.849776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:11.849776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:11.849785 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:11.851550 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:11.852102 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:11.852104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:11.852104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:11.852195 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:11.852519 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:11.852520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:11.852520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:11.852662 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:11.852753 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:11.853187 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 23:15:11.853304 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:15:11.854076 29929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:11.836446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:36753 (local address 127.16.79.3:52999)
0620 23:15:11.840812 (+ 4366us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:11.840860 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:11.841505 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:11.841810 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:11.841815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:11.842057 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:11.842337 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:11.842343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.843600 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.843602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:11.845105 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:11.845108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.845889 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.845891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:11.846097 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:11.846711 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:11.846721 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:11.849646 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:11.851656 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:11.851663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:11.851665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:11.851986 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:11.852272 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:11.852273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:11.852273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:11.852419 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:11.852790 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:11.852796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:11.852985 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 23:15:11.853616 (+ 631us) client_negotiation.cc:240] Negotiation successful
0620 23:15:11.853809 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4321,"thread_start_us":79,"threads_started":1}
I20260620 23:15:11.854512 29927 heartbeater.cc:347] Connected to a master server at 127.16.79.62:36753
I20260620 23:15:11.854700 29927 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:11.854921 29927 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
W20260620 23:15:11.855535 29927 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:36753 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:15:11.855612 29927 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:11.855679 29927 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
I20260620 23:15:11.855993 29927 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:11.856091 29927 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
W20260620 23:15:11.856407 29927 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:15:11.856886 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:11.846229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.16.79.62:36753)
0620 23:15:11.846910 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:11.846912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:11.846918 (+ 6us) server_negotiation.cc:407] Connection header received
0620 23:15:11.846939 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:11.846939 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:11.846951 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:11.847007 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:11.848577 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.849452 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.851117 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:11.851430 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:11.851484 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:11.852959 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:11.852960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:11.852960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:11.852967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:11.855302 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:11.855803 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:11.855804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:11.855805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:11.855863 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:11.856040 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:11.856041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:11.856041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:11.856237 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:11.856388 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:11.856663 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:15:11.856788 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":71,"threads_started":1}
Jun 20 23:15:12 dist-test-slave-pswb krb5kdc[28465](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997312, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:12.489961 29659 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:12.490177 29659 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
I20260620 23:15:12.490666 29461 ts_manager.cc:194] Registered new tserver with Master: 8fbfefac38b04f72ae981093e9b98f3d (127.16.79.1:44099)
I20260620 23:15:12.491707 29461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:51895
I20260620 23:15:12.681324 29793 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:12.681499 29793 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
I20260620 23:15:12.681981 29461 ts_manager.cc:194] Registered new tserver with Master: 141418c867ce406994e8cae68484bdbf (127.16.79.2:40663)
I20260620 23:15:12.682526 29461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:49517
Jun 20 23:15:12 dist-test-slave-pswb krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997312, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for HTTP/127.16.79.20@KRBTEST.COM
I20260620 23:15:12.856611 29927 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:12.856830 29927 heartbeater.cc:511] Master 127.16.79.62:36753 requested a full tablet report, sending...
I20260620 23:15:12.857120 29461 ts_manager.cc:194] Registered new tserver with Master: caa707d1452f48f1919335976a3916f7 (127.16.79.3:44353)
I20260620 23:15:12.857539 29461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:52999
I20260620 23:15:12.858819 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:15:12.867470 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:12.859883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.16.79.62:36753)
0620 23:15:12.859991 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:12.859993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:12.860002 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:15:12.860025 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:12.860025 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:12.860036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:12.860106 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:12.860753 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.861485 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.862500 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.862815 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.862891 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:12.864370 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:12.864371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:12.864371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:12.864377 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:12.866145 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:12.866575 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:12.866578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:12.866578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:12.866673 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:12.866980 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:12.866982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:12.866982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:12.867110 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:12.867205 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:12.867338 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 23:15:12.867405 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260620 23:15:12.869764 29461 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:15:12.872059 29461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:15:12.879776 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:12.875071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:44099 (local address 127.0.0.1:49206)
0620 23:15:12.875232 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:12.875270 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:12.875337 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:12.876037 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:12.876039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:12.876042 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:12.876109 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:12.876110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.877473 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.877474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:12.878565 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:12.878568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.879484 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.879485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:12.879557 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:12.879567 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:15:12.879627 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 23:15:12.879709 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":108,"threads_started":1}
I20260620 23:15:12.879779 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:12.875336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.16.79.1:44099)
0620 23:15:12.875808 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:12.875812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:12.875822 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:15:12.875856 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:12.875858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:12.875874 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:12.875949 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:12.876198 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.877357 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.878646 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.879380 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.879466 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:12.879539 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:12.879654 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 23:15:12.879699 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":62,"threads_started":1}
I20260620 23:15:12.880546 29593 tablet_service.cc:1467] Processing CreateTablet for tablet bd9c5b20c37d49289e82d2d00c4a0abc (DEFAULT_TABLE table=test-table [id=cc6615c571914da09dcce0d2a4215e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:12.880949 29593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9c5b20c37d49289e82d2d00c4a0abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:12.885911 29960 tablet_bootstrap.cc:492] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d: Bootstrap starting.
I20260620 23:15:12.886837 29960 tablet_bootstrap.cc:654] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:12.887249 29960 log.cc:824] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:12.888391 29960 tablet_bootstrap.cc:492] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d: No bootstrap required, opened a new log
I20260620 23:15:12.888468 29960 ts_tablet_manager.cc:1397] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:15:12.888515 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:12.875074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:40663 (local address 127.0.0.1:54728)
0620 23:15:12.875616 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:12.875627 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:12.875676 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:12.884355 (+ 8679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:12.884356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:12.884358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:12.884429 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:12.884431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.885725 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.885726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:12.886967 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:12.886971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.888203 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.888204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:12.888271 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:12.888278 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:15:12.888351 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:15:12.888414 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":81,"threads_started":1}
I20260620 23:15:12.888943 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:12.875173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.16.79.2:40663)
0620 23:15:12.884097 (+ 8924us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:12.884102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:12.884114 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:15:12.884155 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:12.884157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:12.884173 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:12.884352 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:12.884527 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.885592 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.887105 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.887981 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.888076 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:12.888164 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:12.888737 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 23:15:12.888801 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8881,"thread_start_us":52,"threads_started":1}
I20260620 23:15:12.888960 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:12.875307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:44353 (local address 127.0.0.1:37626)
0620 23:15:12.883641 (+ 8334us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:12.883657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:12.883721 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:12.884076 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:12.884077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:12.884079 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:12.884134 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:12.884136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.885797 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.885798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:12.887540 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:12.887543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.888645 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.888646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:12.888751 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:12.888755 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:15:12.888812 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 23:15:12.888864 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8315,"mutex_wait_us":62,"thread_start_us":58,"threads_started":1}
I20260620 23:15:12.889278 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:12.875480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.16.79.3:44353)
0620 23:15:12.875607 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:12.875610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:12.883804 (+ 8194us) server_negotiation.cc:407] Connection header received
0620 23:15:12.883857 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:12.883859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:12.883879 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:12.883961 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:12.884248 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.885664 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.887642 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.888426 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.888510 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:12.888604 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:12.889152 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 23:15:12.889198 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260620 23:15:12.889827 29727 tablet_service.cc:1467] Processing CreateTablet for tablet bd9c5b20c37d49289e82d2d00c4a0abc (DEFAULT_TABLE table=test-table [id=cc6615c571914da09dcce0d2a4215e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:12.889982 29861 tablet_service.cc:1467] Processing CreateTablet for tablet bd9c5b20c37d49289e82d2d00c4a0abc (DEFAULT_TABLE table=test-table [id=cc6615c571914da09dcce0d2a4215e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:12.890239 29727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9c5b20c37d49289e82d2d00c4a0abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:12.890408 29861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9c5b20c37d49289e82d2d00c4a0abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:12.891180 29960 raft_consensus.cc:348] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141418c867ce406994e8cae68484bdbf" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40663 } } peers { permanent_uuid: "8fbfefac38b04f72ae981093e9b98f3d" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44099 } } peers { permanent_uuid: "caa707d1452f48f1919335976a3916f7" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44353 } }
I20260620 23:15:12.891369 29960 raft_consensus.cc:374] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:12.891407 29960 raft_consensus.cc:729] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fbfefac38b04f72ae981093e9b98f3d, State: Initialized, Role: FOLLOWER
I20260620 23:15:12.891546 29960 consensus_queue.cc:260] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141418c867ce406994e8cae68484bdbf" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40663 } } peers { permanent_uuid: "8fbfefac38b04f72ae981093e9b98f3d" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44099 } } peers { permanent_uuid: "caa707d1452f48f1919335976a3916f7" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44353 } }
I20260620 23:15:12.891885 29960 ts_tablet_manager.cc:1428] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:15:12.892745 29659 heartbeater.cc:503] Master 127.16.79.62:36753 was elected leader, sending a full tablet report...
I20260620 23:15:12.894579 29963 tablet_bootstrap.cc:492] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf: Bootstrap starting.
I20260620 23:15:12.895231 29963 tablet_bootstrap.cc:654] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:12.895522 29963 log.cc:824] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:12.895850 29964 tablet_bootstrap.cc:492] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7: Bootstrap starting.
I20260620 23:15:12.896493 29964 tablet_bootstrap.cc:654] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:12.896855 29964 log.cc:824] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:12.897536 29963 tablet_bootstrap.cc:492] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf: No bootstrap required, opened a new log
I20260620 23:15:12.897616 29963 ts_tablet_manager.cc:1397] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 23:15:12.899441 29964 tablet_bootstrap.cc:492] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7: No bootstrap required, opened a new log
I20260620 23:15:12.899569 29964 ts_tablet_manager.cc:1397] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:15:12.900408 29963 raft_consensus.cc:348] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141418c867ce406994e8cae68484bdbf" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40663 } } peers { permanent_uuid: "8fbfefac38b04f72ae981093e9b98f3d" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44099 } } peers { permanent_uuid: "caa707d1452f48f1919335976a3916f7" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44353 } }
I20260620 23:15:12.900640 29963 raft_consensus.cc:374] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:12.900760 29963 raft_consensus.cc:729] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141418c867ce406994e8cae68484bdbf, State: Initialized, Role: FOLLOWER
I20260620 23:15:12.900951 29963 consensus_queue.cc:260] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141418c867ce406994e8cae68484bdbf" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40663 } } peers { permanent_uuid: "8fbfefac38b04f72ae981093e9b98f3d" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44099 } } peers { permanent_uuid: "caa707d1452f48f1919335976a3916f7" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44353 } }
I20260620 23:15:12.901311 29963 ts_tablet_manager.cc:1428] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 23:15:12.901473 29793 heartbeater.cc:503] Master 127.16.79.62:36753 was elected leader, sending a full tablet report...
I20260620 23:15:12.902853 29964 raft_consensus.cc:348] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141418c867ce406994e8cae68484bdbf" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40663 } } peers { permanent_uuid: "8fbfefac38b04f72ae981093e9b98f3d" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44099 } } peers { permanent_uuid: "caa707d1452f48f1919335976a3916f7" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44353 } }
I20260620 23:15:12.903012 29964 raft_consensus.cc:374] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:12.903049 29964 raft_consensus.cc:729] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa707d1452f48f1919335976a3916f7, State: Initialized, Role: FOLLOWER
I20260620 23:15:12.903214 29964 consensus_queue.cc:260] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141418c867ce406994e8cae68484bdbf" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40663 } } peers { permanent_uuid: "8fbfefac38b04f72ae981093e9b98f3d" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44099 } } peers { permanent_uuid: "caa707d1452f48f1919335976a3916f7" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44353 } }
I20260620 23:15:12.903462 29964 ts_tablet_manager.cc:1428] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:15:12.903497 29927 heartbeater.cc:503] Master 127.16.79.62:36753 was elected leader, sending a full tablet report...
W20260620 23:15:12.919497 29794 tablet.cc:2369] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:12.968791 29968 raft_consensus.cc:482] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:12.968946 29968 raft_consensus.cc:504] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141418c867ce406994e8cae68484bdbf" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40663 } } peers { permanent_uuid: "8fbfefac38b04f72ae981093e9b98f3d" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44099 } } peers { permanent_uuid: "caa707d1452f48f1919335976a3916f7" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44353 } }
I20260620 23:15:12.969390 29968 leader_election.cc:288] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 141418c867ce406994e8cae68484bdbf (127.16.79.2:40663), 8fbfefac38b04f72ae981093e9b98f3d (127.16.79.1:44099)
I20260620 23:15:12.973698 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:12.969798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:60485 (local address 127.16.79.2:40663)
0620 23:15:12.969843 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:12.969846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:12.969996 (+ 150us) server_negotiation.cc:407] Connection header received
0620 23:15:12.970111 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:12.970112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:12.970123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:12.970152 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:12.970380 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.971133 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.972670 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.973300 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.973350 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:12.973363 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:12.973578 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 23:15:12.973623 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:15:12.974130 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:12.969613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:40663 (local address 127.16.79.3:60485)
0620 23:15:12.969929 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:12.969940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:12.970036 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:12.970215 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:12.970216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:12.970217 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:12.970310 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:12.970312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.971233 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.971234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:12.972564 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:12.972567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.973424 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.973425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:12.973495 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:12.973499 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:15:12.973986 (+ 487us) client_negotiation.cc:240] Negotiation successful
0620 23:15:12.974045 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":35,"thread_start_us":88,"threads_started":1}
W20260620 23:15:12.975493 29660 tablet.cc:2369] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:12.975984 29747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9c5b20c37d49289e82d2d00c4a0abc" candidate_uuid: "caa707d1452f48f1919335976a3916f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141418c867ce406994e8cae68484bdbf" is_pre_election: true
I20260620 23:15:12.976166 29747 raft_consensus.cc:2454] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caa707d1452f48f1919335976a3916f7 in term 0.
I20260620 23:15:12.976500 29817 leader_election.cc:302] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 141418c867ce406994e8cae68484bdbf, caa707d1452f48f1919335976a3916f7; no voters:
I20260620 23:15:12.976684 29968 raft_consensus.cc:2781] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:15:12.976770 29968 raft_consensus.cc:482] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:15:12.976794 29968 raft_consensus.cc:3037] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:12.977700 29968 raft_consensus.cc:504] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141418c867ce406994e8cae68484bdbf" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40663 } } peers { permanent_uuid: "8fbfefac38b04f72ae981093e9b98f3d" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44099 } } peers { permanent_uuid: "caa707d1452f48f1919335976a3916f7" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44353 } }
I20260620 23:15:12.977986 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:12.969796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:34465 (local address 127.16.79.1:44099)
0620 23:15:12.969843 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:12.969845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:12.969974 (+ 129us) server_negotiation.cc:407] Connection header received
0620 23:15:12.970083 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:12.970084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:12.970110 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:12.970141 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:12.970410 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.971093 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.976937 (+ 5844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.977542 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.977594 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:12.977604 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:12.977879 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:15:12.977920 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:15:12.978245 29970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:12.969581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:44099 (local address 127.16.79.3:34465)
0620 23:15:12.969906 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:12.969918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:12.970005 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:12.970217 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:12.970217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:12.970219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:12.970323 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:12.970325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.971209 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.971210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:12.976836 (+ 5626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:12.976838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:12.977682 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:12.977683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:12.977790 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:12.977794 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:15:12.978103 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 23:15:12.978147 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":76,"thread_start_us":98,"threads_started":1}
I20260620 23:15:12.978231 29747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9c5b20c37d49289e82d2d00c4a0abc" candidate_uuid: "caa707d1452f48f1919335976a3916f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141418c867ce406994e8cae68484bdbf"
I20260620 23:15:12.978302 29747 raft_consensus.cc:3037] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:12.978590 29968 leader_election.cc:288] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [CANDIDATE]: Term 1 election: Requested vote from peers 141418c867ce406994e8cae68484bdbf (127.16.79.2:40663), 8fbfefac38b04f72ae981093e9b98f3d (127.16.79.1:44099)
I20260620 23:15:12.978662 29613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9c5b20c37d49289e82d2d00c4a0abc" candidate_uuid: "caa707d1452f48f1919335976a3916f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fbfefac38b04f72ae981093e9b98f3d" is_pre_election: true
I20260620 23:15:12.978796 29613 raft_consensus.cc:2454] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caa707d1452f48f1919335976a3916f7 in term 0.
I20260620 23:15:12.978899 29612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9c5b20c37d49289e82d2d00c4a0abc" candidate_uuid: "caa707d1452f48f1919335976a3916f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fbfefac38b04f72ae981093e9b98f3d"
I20260620 23:15:12.978955 29612 raft_consensus.cc:3037] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:12.979161 29747 raft_consensus.cc:2454] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caa707d1452f48f1919335976a3916f7 in term 1.
I20260620 23:15:12.979391 29817 leader_election.cc:302] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 141418c867ce406994e8cae68484bdbf, caa707d1452f48f1919335976a3916f7; no voters:
I20260620 23:15:12.979461 29968 raft_consensus.cc:2781] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:12.979593 29968 raft_consensus.cc:686] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 1 LEADER]: Becoming Leader. State: Replica: caa707d1452f48f1919335976a3916f7, State: Running, Role: LEADER
I20260620 23:15:12.979696 29968 consensus_queue.cc:237] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141418c867ce406994e8cae68484bdbf" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40663 } } peers { permanent_uuid: "8fbfefac38b04f72ae981093e9b98f3d" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44099 } } peers { permanent_uuid: "caa707d1452f48f1919335976a3916f7" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44353 } }
I20260620 23:15:12.979889 29612 raft_consensus.cc:2454] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caa707d1452f48f1919335976a3916f7 in term 1.
I20260620 23:15:12.980613 29461 catalog_manager.cc:5495] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to caa707d1452f48f1919335976a3916f7 (127.16.79.3). New cstate: current_term: 1 leader_uuid: "caa707d1452f48f1919335976a3916f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141418c867ce406994e8cae68484bdbf" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fbfefac38b04f72ae981093e9b98f3d" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "caa707d1452f48f1919335976a3916f7" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44353 } health_report { overall_health: HEALTHY } } }
I20260620 23:15:13.003957 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:13.000958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37640 (local address 127.16.79.3:44353)
0620 23:15:13.001084 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:13.001085 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:13.001091 (+ 6us) server_negotiation.cc:407] Connection header received
0620 23:15:13.001127 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:13.001128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:13.001145 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:13.001372 (+ 227us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:13.001648 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:13.002293 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:13.003129 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:13.003348 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:13.003436 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:13.003617 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:13.003721 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:15:13.003816 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:13.003836 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 23:15:13.003875 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 23:15:13.006254 29612 raft_consensus.cc:1261] T bd9c5b20c37d49289e82d2d00c4a0abc P 8fbfefac38b04f72ae981093e9b98f3d [term 1 FOLLOWER]: Refusing update from remote peer caa707d1452f48f1919335976a3916f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:13.006685 29971 consensus_queue.cc:1035] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fbfefac38b04f72ae981093e9b98f3d" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 44099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:13.007272 29747 raft_consensus.cc:1261] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf [term 1 FOLLOWER]: Refusing update from remote peer caa707d1452f48f1919335976a3916f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:13.007602 29971 consensus_queue.cc:1035] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "141418c867ce406994e8cae68484bdbf" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:13.010610 29974 mvcc.cc:205] Tried to move back new op lower bound from 7299060994070581248 to 7299060993965596672. Current Snapshot: MvccSnapshot[applied={T|T < 7299060994070581248}]
I20260620 23:15:13.014611 29461 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:13.014698 29461 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:13.015964 29461 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9cc4074d37846048e24fe3692004be4: Sending DeleteTablet for 3 replicas of tablet bd9c5b20c37d49289e82d2d00c4a0abc
I20260620 23:15:13.016533 29593 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9c5b20c37d49289e82d2d00c4a0abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:13 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:49206
I20260620 23:15:13.016827 29861 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9c5b20c37d49289e82d2d00c4a0abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:13 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:37626
I20260620 23:15:13.017427 29727 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9c5b20c37d49289e82d2d00c4a0abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:13 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:54728
I20260620 23:15:13.017760 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 29499
I20260620 23:15:13.011300 29977 mvcc.cc:205] Tried to move back new op lower bound from 7299060994070581248 to 7299060993965596672. Current Snapshot: MvccSnapshot[applied={T|T < 7299060994070581248}]
I20260620 23:15:13.024801 29984 tablet_replica.cc:330] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7: stopping tablet replica
I20260620 23:15:13.024976 29984 raft_consensus.cc:2229] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:15:13.025120 29984 raft_consensus.cc:2258] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:13.028797 29985 tablet_replica.cc:330] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf: stopping tablet replica
I20260620 23:15:13.028977 29985 raft_consensus.cc:2229] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:13.029139 29985 raft_consensus.cc:2258] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:13.029783 29984 ts_tablet_manager.cc:1905] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:13.032359 29984 ts_tablet_manager.cc:1918] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:13.032498 29984 log.cc:1196] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/bd9c5b20c37d49289e82d2d00c4a0abc
I20260620 23:15:13.032915 29984 ts_tablet_manager.cc:1939] T bd9c5b20c37d49289e82d2d00c4a0abc P caa707d1452f48f1919335976a3916f7: Deleting consensus metadata
I20260620 23:15:13.033461 29445 catalog_manager.cc:4841] TS caa707d1452f48f1919335976a3916f7 (127.16.79.3:44353): tablet bd9c5b20c37d49289e82d2d00c4a0abc (table test-table [id=cc6615c571914da09dcce0d2a4215e0a]) successfully deleted
I20260620 23:15:13.033821 29985 ts_tablet_manager.cc:1905] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:13.035686 29985 ts_tablet_manager.cc:1918] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:13.035823 29985 log.cc:1196] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/bd9c5b20c37d49289e82d2d00c4a0abc
I20260620 23:15:13.036165 29985 ts_tablet_manager.cc:1939] T bd9c5b20c37d49289e82d2d00c4a0abc P 141418c867ce406994e8cae68484bdbf: Deleting consensus metadata
I20260620 23:15:13.036612 29448 catalog_manager.cc:4841] TS 141418c867ce406994e8cae68484bdbf (127.16.79.2:40663): tablet bd9c5b20c37d49289e82d2d00c4a0abc (table test-table [id=cc6615c571914da09dcce0d2a4215e0a]) successfully deleted
W20260620 23:15:13.036969 29447 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:44099 (error 108)
I20260620 23:15:13.044822 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 29663
I20260620 23:15:13.045576 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:13.044958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:44099 (local address 127.0.0.1:49210)
0620 23:15:13.045497 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:13.045542 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516}
W20260620 23:15:13.045830 29447 catalog_manager.cc:4584] TS 8fbfefac38b04f72ae981093e9b98f3d (127.16.79.1:44099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd9c5b20c37d49289e82d2d00c4a0abc: Network error: Client connection negotiation failed: client connection to 127.16.79.1:44099: connect: Connection refused (error 111)
I20260620 23:15:13.049991 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 29796
I20260620 23:15:13.056655 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 29428
I20260620 23:15:13.064944 16700 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:15:13.409765 16700 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:15:13.409902 16700 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:15:13.754444 16700 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:15:13.754 UTC [28495] LOG: received smart shutdown request
2026-06-20 23:15:13.758 UTC [28495] LOG: background worker "logical replication launcher" (PID 28504) exited with exit code 1
2026-06-20 23:15:13.758 UTC [28499] LOG: shutting down
2026-06-20 23:15:13.942 UTC [28495] LOG: database system is shut down
2026-06-20T23:15:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47305 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:15:14 dist-test-slave-pswb krb5kdc[29996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:15:14 dist-test-slave-pswb krb5kdc[29996](info): set up 2 sockets
Jun 20 23:15:14 dist-test-slave-pswb krb5kdc[29996](info): commencing operation
krb5kdc: starting...
W20260620 23:15:15.104748 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:15:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:15:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:15:15.226842 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:46509
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:46509,127.16.79.61:35349,127.16.79.60:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:15.311161 30009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:15.311367 30009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:15.311391 30009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:15.312505 30009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:15.312557 30009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:15.312578 30009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:15.312599 30009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:15.312619 30009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:15.314146 30009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:46509,127.16.79.61:35349,127.16.79.60:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:46509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:15.314349 30009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:15.314519 30009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:15.319175 30015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:15.319175 30017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:15.319239 30014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:15.319466 30009 server_base.cc:1034] running on GCE node
I20260620 23:15:15.319700 30009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:15.319967 30009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:15.321156 30009 hybrid_clock.cc:648] HybridClock initialized: now 1781997315321129 us; error 40 us; skew 500 ppm
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:15.327567 30009 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:15:15.328248 30009 webserver.cc:460] Webserver started at http://127.16.79.62:45001/ using document root <none> and password file <none>
I20260620 23:15:15.328569 30009 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:15.328644 30009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:15.328833 30009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:15.330035 30009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "ac97ef9234734f03a835cc5fe61ac2ee"
format_stamp: "Formatted at 2026-06-20 23:15:15 on dist-test-slave-pswb"
server_key: "72f58d425851949190fa034db2c6ba9a"
server_key_iv: "d4d255bd9f704a5269f752dcdc803344"
server_key_version: "encryptionkey@0"
I20260620 23:15:15.330456 30009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "ac97ef9234734f03a835cc5fe61ac2ee"
format_stamp: "Formatted at 2026-06-20 23:15:15 on dist-test-slave-pswb"
server_key: "72f58d425851949190fa034db2c6ba9a"
server_key_iv: "d4d255bd9f704a5269f752dcdc803344"
server_key_version: "encryptionkey@0"
I20260620 23:15:15.332096 30009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:15:15.333302 30024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:15.333746 30009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:15.333889 30009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "ac97ef9234734f03a835cc5fe61ac2ee"
format_stamp: "Formatted at 2026-06-20 23:15:15 on dist-test-slave-pswb"
server_key: "72f58d425851949190fa034db2c6ba9a"
server_key_iv: "d4d255bd9f704a5269f752dcdc803344"
server_key_version: "encryptionkey@0"
I20260620 23:15:15.333986 30009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:15.353868 30009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:15.356144 30009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:15.356304 30009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:15.361053 30009 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:46509
I20260620 23:15:15.361095 30076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:46509 every 8 connection(s)
I20260620 23:15:15.361634 30009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:15.361838 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 30009
I20260620 23:15:15.361958 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:15:15.362205 16700 external_mini_cluster.cc:1428] Setting key 58dfa768727bbebbbad0296798ec90b0
I20260620 23:15:15.362756 30077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:15.364663 30077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.79.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.61@KRBTEST.COM" created.
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:15.377629 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.365742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46509 (local address 127.0.0.1:49858)
0620 23:15:15.365965 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.366016 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:15.366671 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:15.367082 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:15.367085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:15.367339 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:15.367489 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.367493 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.369086 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.369087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.369872 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.369876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.370407 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.370408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.370463 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.370987 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:15.370998 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:15.373770 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:15.375697 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.375710 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.375712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.375917 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.376207 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.376208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.376208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.376318 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.376844 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:15.376859 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:15.376969 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 23:15:15.377097 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 23:15:15.377475 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":144,"thread_start_us":125,"threads_started":1}
I20260620 23:15:15.377641 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.366032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.16.79.62:46509)
0620 23:15:15.366230 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.366236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.366280 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:15:15.366868 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.366871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.366899 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.367041 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.367602 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.368970 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.369952 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.370326 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.370410 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.373863 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.373870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.373871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.373878 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.375593 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.376008 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.376011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.376012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.376117 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.376389 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.376390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.376391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.376714 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.376804 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.377180 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.377338 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260620 23:15:15.378515 30077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:15:15.378855 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.378747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:35349 (local address 127.0.0.1:48564)
0620 23:15:15.378793 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.378836 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 23:15:15.379016 30027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.79.61:35349: connect: Connection refused (error 111)
I20260620 23:15:15.379422 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.379330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:35349 (local address 127.0.0.1:48574)
0620 23:15:15.379373 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.379396 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 23:15:15.379618 30077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.61:35349: Network error: Client connection negotiation failed: client connection to 127.16.79.61:35349: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.61.keytab.
Entry for principal kudu/127.16.79.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.61.keytab.
I20260620 23:15:15.392872 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.61:35349
--webserver_interface=127.16.79.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.61.keytab
--principal=kudu/127.16.79.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:46509,127.16.79.61:35349,127.16.79.60:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:15.428830 30077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } attempt: 1
I20260620 23:15:15.429337 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.429207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:35349 (local address 127.0.0.1:48582)
0620 23:15:15.429270 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.429309 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 23:15:15.429706 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.429624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:35349 (local address 127.0.0.1:48596)
0620 23:15:15.429666 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.429690 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 23:15:15.429811 30077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.61:35349: Network error: Client connection negotiation failed: client connection to 127.16.79.61:35349: connect: Connection refused (error 111)
W20260620 23:15:15.477618 30084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:15.477818 30084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:15.477841 30084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:15.478950 30084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:15.478989 30084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:15.479003 30084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:15.479017 30084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:15.479028 30084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:15.480553 30084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:46509,127.16.79.61:35349,127.16.79.60:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.61.keytab
--principal=kudu/127.16.79.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.61:35349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.79.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:15.480847 30084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:15.481082 30084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:15:15.485922 30084 server_base.cc:1034] running on GCE node
W20260620 23:15:15.485924 30089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:15.485908 30090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:15.485908 30092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:15.486415 30084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:15.486644 30084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:15.487779 30084 hybrid_clock.cc:648] HybridClock initialized: now 1781997315487758 us; error 36 us; skew 500 ppm
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:15.490291 30084 init.cc:377] Logged in from keytab as kudu/127.16.79.61@KRBTEST.COM (short username kudu)
I20260620 23:15:15.490777 30084 webserver.cc:460] Webserver started at http://127.16.79.61:43569/ using document root <none> and password file <none>
I20260620 23:15:15.491012 30084 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:15.491065 30084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:15.491192 30084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:15.492249 30084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/data/instance:
uuid: "c7d3e95ec5a74089b3a31d528955547a"
format_stamp: "Formatted at 2026-06-20 23:15:15 on dist-test-slave-pswb"
server_key: "b2cfa889d555a749a66b65e00e91e73d"
server_key_iv: "38e55e91cc886c2fbb10f8ad8b4f4fbc"
server_key_version: "encryptionkey@0"
I20260620 23:15:15.492616 30084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/wal/instance:
uuid: "c7d3e95ec5a74089b3a31d528955547a"
format_stamp: "Formatted at 2026-06-20 23:15:15 on dist-test-slave-pswb"
server_key: "b2cfa889d555a749a66b65e00e91e73d"
server_key_iv: "38e55e91cc886c2fbb10f8ad8b4f4fbc"
server_key_version: "encryptionkey@0"
I20260620 23:15:15.494129 30084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:15.495386 30099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:15.495802 30084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 23:15:15.495949 30084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/wal
uuid: "c7d3e95ec5a74089b3a31d528955547a"
format_stamp: "Formatted at 2026-06-20 23:15:15 on dist-test-slave-pswb"
server_key: "b2cfa889d555a749a66b65e00e91e73d"
server_key_iv: "38e55e91cc886c2fbb10f8ad8b4f4fbc"
server_key_version: "encryptionkey@0"
I20260620 23:15:15.496049 30084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:15.497960 30077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } attempt: 2
I20260620 23:15:15.498327 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.498212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:35349 (local address 127.0.0.1:48598)
0620 23:15:15.498267 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.498299 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 23:15:15.498644 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.498558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:35349 (local address 127.0.0.1:48612)
0620 23:15:15.498599 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.498622 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 23:15:15.498728 30077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.61:35349: Network error: Client connection negotiation failed: client connection to 127.16.79.61:35349: connect: Connection refused (error 111)
I20260620 23:15:15.513235 30084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:15.515254 30084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:15.515416 30084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:15.521067 30084 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.61:35349
I20260620 23:15:15.521173 30151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.61:35349 every 8 connection(s)
I20260620 23:15:15.521603 30084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
I20260620 23:15:15.522526 30152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:15.524065 30152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:15:15.528183 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 30084
I20260620 23:15:15.528383 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-1/wal/instance
I20260620 23:15:15.528649 16700 external_mini_cluster.cc:1428] Setting key 98e582a3ff7f8d638c414fca24bbcd17
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.61@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:15.535156 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.524815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.16.79.62:46509)
0620 23:15:15.524887 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.524890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.525164 (+ 274us) server_negotiation.cc:407] Connection header received
0620 23:15:15.525746 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.525747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.525761 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.525823 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.526607 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.527087 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.528660 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.529017 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.529097 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.531652 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.531653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.531654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.531661 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.533308 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.533889 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.533891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.533891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.534002 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.534403 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.534404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.534404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.534558 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.534623 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.535015 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.535090 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:15:15.535377 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.524814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46509 (local address 127.0.0.1:49872)
0620 23:15:15.525040 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.525075 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:15.525483 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:15.526001 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:15.526004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:15.526241 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:15.526424 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.526427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.527292 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.527294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.528468 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.528472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.529159 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.529161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.529302 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.529897 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:15.529908 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:15.531557 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:15.533403 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.533411 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.533414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.533748 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.534135 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.534136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.534137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.534288 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.534640 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:15.534648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:15.534816 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 23:15:15.534946 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 23:15:15.535131 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":118,"threads_started":1}
I20260620 23:15:15.536060 30152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.79.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.60@KRBTEST.COM" created.
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.61@KRBTEST.COM for kudu/127.16.79.61@KRBTEST.COM
I20260620 23:15:15.544585 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.536429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48620 (local address 127.16.79.61:35349)
0620 23:15:15.536628 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.536631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.536647 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:15:15.536663 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.536664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.536675 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.536770 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.537077 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.537884 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.538945 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.539366 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.539444 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.541521 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.541529 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.541530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.541538 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.543177 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.543632 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.543634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.543634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.543685 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.543917 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.543917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.543918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.544042 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.544108 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.544351 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.544451 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260620 23:15:15.544636 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.536359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:35349 (local address 127.0.0.1:48620)
0620 23:15:15.536432 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.536442 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:15.536481 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:15.536797 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:15.536798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:15.536943 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:15.536998 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.537000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.538077 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.538080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.538757 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.538759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.539528 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.539530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.539666 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.540107 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:15.540113 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:15.541428 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:15.543272 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.543273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.543274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.543533 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.543766 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.543767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.543768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.543845 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.544121 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:15.544122 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:15.544177 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 23:15:15.544269 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:15:15.544394 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:15:15.545632 30152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:15:15.546007 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.545859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40088)
0620 23:15:15.545916 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.545979 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 23:15:15.546201 30100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
I20260620 23:15:15.546700 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.546581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40100)
0620 23:15:15.546640 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.546673 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 23:15:15.546823 30152 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:46753: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.60.keytab.
Entry for principal kudu/127.16.79.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.60.keytab.
I20260620 23:15:15.556470 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.60:46753
--webserver_interface=127.16.79.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.60.keytab
--principal=kudu/127.16.79.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:46509,127.16.79.61:35349,127.16.79.60:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:15.589860 30077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } attempt: 3
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for kudu/127.16.79.61@KRBTEST.COM
I20260620 23:15:15.596037 30152 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } attempt: 1
I20260620 23:15:15.596414 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.596290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40112)
0620 23:15:15.596341 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.596372 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 23:15:15.596967 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.596851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40128)
0620 23:15:15.596910 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.596937 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:15:15.597131 30152 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:46753: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
I20260620 23:15:15.597878 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.590346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.16.79.61:35349)
0620 23:15:15.590409 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.590411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.590421 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:15:15.590446 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.590446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.590457 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.590520 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.590986 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.591569 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.592715 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.592961 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.593022 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.595076 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.595082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.595082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.595089 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.596740 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.597125 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.597126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.597126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.597210 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.597438 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.597439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.597439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.597550 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.597599 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.597768 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.597828 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:15:15.597945 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.590272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:35349 (local address 127.0.0.1:48624)
0620 23:15:15.590334 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.590346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:15.590388 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:15.590544 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:15.590545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:15.590727 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:15.590802 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.590804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.591684 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.591685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.592561 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.592564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.593046 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.593048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.593130 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.593631 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:15.593637 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:15.594976 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:15.596818 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.596818 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.596819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.597030 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.597279 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.597280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.597280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.597366 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.597617 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:15.597619 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:15.597673 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 23:15:15.597754 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:15:15.597850 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 23:15:15.598390 30077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:15:15.598714 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.598614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40140)
0620 23:15:15.598659 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.598689 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 23:15:15.599045 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.598950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40154)
0620 23:15:15.598995 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.599020 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 23:15:15.599203 30077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:46753: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
I20260620 23:15:15.630353 30077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } attempt: 1
I20260620 23:15:15.630795 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.630662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40162)
0620 23:15:15.630722 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.630762 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 23:15:15.631170 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.631063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40172)
0620 23:15:15.631111 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.631140 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 23:15:15.631317 30077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:46753: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
W20260620 23:15:15.644745 30159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:15.644965 30159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:15.645010 30159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:15.646113 30159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:15.646152 30159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:15.646166 30159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:15.646179 30159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:15.646230 30159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:15.647657 30159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:46509,127.16.79.61:35349,127.16.79.60:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.60.keytab
--principal=kudu/127.16.79.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.60:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.79.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:15.647895 30159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:15.648136 30159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:15.652825 30164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:15.652848 30167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:15.652856 30165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:15.653173 30159 server_base.cc:1034] running on GCE node
I20260620 23:15:15.653430 30159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:15.653704 30159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:15.654855 30159 hybrid_clock.cc:648] HybridClock initialized: now 1781997315654846 us; error 30 us; skew 500 ppm
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:15.657323 30159 init.cc:377] Logged in from keytab as kudu/127.16.79.60@KRBTEST.COM (short username kudu)
I20260620 23:15:15.657684 30159 webserver.cc:460] Webserver started at http://127.16.79.60:44019/ using document root <none> and password file <none>
I20260620 23:15:15.657903 30159 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:15.657972 30159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:15.658100 30159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:15.659126 30159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/data/instance:
uuid: "7f1ae3720cc643a1b6fbc55547863d5c"
format_stamp: "Formatted at 2026-06-20 23:15:15 on dist-test-slave-pswb"
server_key: "7e253c12ed1f13fd0c30284e9916a725"
server_key_iv: "5b46ffc0438627c1533cda26ac7f32da"
server_key_version: "encryptionkey@0"
I20260620 23:15:15.659489 30159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/wal/instance:
uuid: "7f1ae3720cc643a1b6fbc55547863d5c"
format_stamp: "Formatted at 2026-06-20 23:15:15 on dist-test-slave-pswb"
server_key: "7e253c12ed1f13fd0c30284e9916a725"
server_key_iv: "5b46ffc0438627c1533cda26ac7f32da"
server_key_version: "encryptionkey@0"
I20260620 23:15:15.660871 30159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:15:15.661648 30174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:15.661851 30159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:15.661962 30159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/wal
uuid: "7f1ae3720cc643a1b6fbc55547863d5c"
format_stamp: "Formatted at 2026-06-20 23:15:15 on dist-test-slave-pswb"
server_key: "7e253c12ed1f13fd0c30284e9916a725"
server_key_iv: "5b46ffc0438627c1533cda26ac7f32da"
server_key_version: "encryptionkey@0"
I20260620 23:15:15.662050 30159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:15.665299 30152 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } attempt: 2
I20260620 23:15:15.665701 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.665574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40174)
0620 23:15:15.665640 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.665673 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 23:15:15.666079 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.665978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40180)
0620 23:15:15.666010 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.666035 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 23:15:15.666702 30152 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:46753: Network error: Client connection negotiation failed: client connection to 127.16.79.60:46753: connect: Connection refused (error 111)
I20260620 23:15:15.683043 30159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:15.685493 30159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:15.685603 30159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:15.689515 30159 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.60:46753
I20260620 23:15:15.689565 30226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.60:46753 every 8 connection(s)
I20260620 23:15:15.689929 30159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
I20260620 23:15:15.690773 30227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:15.692348 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 30159
I20260620 23:15:15.692276 30227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:15:15.692564 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997163808439-16700-0/minicluster-data/master-2/wal/instance
I20260620 23:15:15.692821 16700 external_mini_cluster.cc:1428] Setting key 540f1638c73539d7261a0264b33c8d0f
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.60@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.60@KRBTEST.COM
I20260620 23:15:15.703213 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.693092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49888 (local address 127.16.79.62:46509)
0620 23:15:15.693178 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.693180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.693368 (+ 188us) server_negotiation.cc:407] Connection header received
0620 23:15:15.693907 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.693908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.693919 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.693972 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.694743 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.695610 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.696854 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.697096 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.697175 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.699520 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.699521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.699522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.699529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.701302 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.701760 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.701762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.701762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.701822 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.702260 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.702263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.702263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.702525 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.702620 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.702948 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.703080 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:15:15.703248 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.693065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:46509 (local address 127.0.0.1:49888)
0620 23:15:15.693209 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.693247 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:15.693793 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:15.694008 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:15.694011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:15.694350 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:15.694571 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.694576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.695723 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.695725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.696769 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.696772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.697198 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.697199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.697311 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.697799 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:15.697809 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:15.699430 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:15.701391 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.701397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.701398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.701684 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.701921 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.701921 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.701922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.702069 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.702615 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:15.702621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:15.702767 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 23:15:15.702857 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:15:15.702974 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260620 23:15:15.703559 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.693422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.16.79.60:46753)
0620 23:15:15.693553 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.693557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.693585 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:15:15.693805 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.693814 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.693834 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.693948 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.694768 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.696007 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.697003 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.697387 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.697438 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.700001 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.700009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.700010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.700017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.701840 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.702302 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.702308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.702308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.702399 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.702704 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.702705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.702705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.702945 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.703163 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.703214 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.703413 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260620 23:15:15.704540 30227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:15:15.706496 30077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } attempt: 2
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.60@KRBTEST.COM for kudu/127.16.79.61@KRBTEST.COM
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for kudu/127.16.79.60@KRBTEST.COM
I20260620 23:15:15.712334 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.704381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.16.79.62:46509)
0620 23:15:15.704423 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.704424 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.704455 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:15:15.704473 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.704473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.704490 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.704530 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.705152 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.705697 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.706391 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.706670 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.706765 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.708953 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.708955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.708955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.708961 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.711031 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.711565 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.711566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.711567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.711624 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.711826 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.711826 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.711827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.711931 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.712109 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.712146 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.712233 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:15:15.712683 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.704907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.16.79.61:35349)
0620 23:15:15.704957 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.704959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.704969 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:15:15.704995 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.704996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.705006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.705075 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.705379 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.705881 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.706568 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.706887 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.706954 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.709855 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.709857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.709857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.709864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.711430 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.711799 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.711800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.711801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.711852 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.712078 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.712079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.712079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.712224 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.712267 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.712484 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.712553 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 23:15:15.712967 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.704837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:35349 (local address 127.0.0.1:48636)
0620 23:15:15.704903 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.704911 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:15.704951 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:15.705079 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:15.705080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:15.705251 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:15.705305 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.705307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.705964 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.705965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.706488 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.706489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.706967 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.706968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.707050 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.707916 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:15.707921 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:15.709778 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:15.711517 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.711518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.711518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.711741 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.711929 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.711930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.711930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.711994 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.712321 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:15.712321 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:15.712364 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 23:15:15.712781 (+ 417us) client_negotiation.cc:240] Negotiation successful
0620 23:15:15.712871 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:15:15.713608 30227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:15:15.714527 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.706725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40210)
0620 23:15:15.706796 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.706805 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:15.706836 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:15.707040 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:15.707041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:15.707200 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:15.707240 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.707242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.707841 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.707842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.708623 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.708626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.709048 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.709049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.709109 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.709545 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:15.709550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:15.710833 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:15.712848 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.712849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.712850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.713201 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.713559 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.713560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.713560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.713623 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.714212 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:15.714214 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:15.714256 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:15:15.714346 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:15:15.714437 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 23:15:15.714540 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.706904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.16.79.60:46753)
0620 23:15:15.706940 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.706942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.706949 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:15:15.706968 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.706969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.706981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.707033 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.707320 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.707763 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.708715 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.708963 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.709030 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.710933 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.710934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.710935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.710941 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.712685 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.713417 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.713418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.713418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.713482 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.713700 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.713701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.713701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.713983 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.714212 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.714393 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.714462 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":20}
I20260620 23:15:15.716353 30077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee: Bootstrap starting.
I20260620 23:15:15.717190 30077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:15.717525 30077 log.cc:824] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.60@KRBTEST.COM for kudu/127.16.79.60@KRBTEST.COM
I20260620 23:15:15.718348 30077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee: No bootstrap required, opened a new log
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.61@KRBTEST.COM
I20260620 23:15:15.720465 30077 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } }
I20260620 23:15:15.720651 30077 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:15.720731 30077 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac97ef9234734f03a835cc5fe61ac2ee, State: Initialized, Role: FOLLOWER
I20260620 23:15:15.720886 30077 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } }
I20260620 23:15:15.721294 30077 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:15.721300 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.713850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40226)
0620 23:15:15.714049 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.714058 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:15.714101 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:15.714280 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:15.714281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:15.714417 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:15.714452 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.714453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.715298 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.715299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.715994 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.715996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.716648 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.716649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.716722 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.717111 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:15.717115 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:15.718395 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:15.720218 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.720219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.720219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.720426 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.720630 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.720631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.720631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.720685 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.720985 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:15.720985 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:15.721023 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:15:15.721112 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:15:15.721202 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260620 23:15:15.721390 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.713897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.16.79.60:46753)
0620 23:15:15.714049 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.714051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.714095 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:15:15.714172 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.714173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.714181 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.714248 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.714531 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.715226 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.716069 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.716290 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.716646 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.718501 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.718501 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.718502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.718507 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.720121 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.720505 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.720506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.720506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.720554 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.720775 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.720775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.720776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.720917 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.720986 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.721135 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.721251 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":134,"thread_start_us":109,"threads_started":1}
I20260620 23:15:15.722678 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.713349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48642 (local address 127.16.79.61:35349)
0620 23:15:15.713435 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.713436 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.713443 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:15:15.713460 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.713461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.713470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.713520 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.714161 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.714854 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.715763 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.716036 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.716105 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.719173 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.719174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.719174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.719181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.721309 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.721765 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.721766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.721767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.721822 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.722034 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.722035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.722035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.722152 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.722303 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.722520 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.722595 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:15:15.722671 30227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c: Bootstrap starting.
I20260620 23:15:15.723340 30227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:15.723628 30227 log.cc:824] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:15.723093 30234 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } } }
I20260620 23:15:15.724251 30234 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:15:15.724542 30227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c: No bootstrap required, opened a new log
I20260620 23:15:15.725493 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:15:15.726619 30227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } }
I20260620 23:15:15.726747 30227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:15.726779 30227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1ae3720cc643a1b6fbc55547863d5c, State: Initialized, Role: FOLLOWER
I20260620 23:15:15.726888 30227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } }
I20260620 23:15:15.727165 30227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:15:15.723500 30245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:15:15.728847 30245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:15:15.728899 30009 master_runner.cc:186] Error getting master registration for 127.16.79.61:35349: OK, instance_id { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" instance_seqno: 1781997315519404 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:15:15.731097 30009 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 23:15:15.732067 30266 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:15:15.732115 30266 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:15:15.732435 30254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } } }
I20260620 23:15:15.732571 30254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:15:15.732515 30159 master_runner.cc:186] Error getting master registration for 127.16.79.61:35349: OK, instance_id { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" instance_seqno: 1781997315519404 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:15:15.735589 30159 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:15:15.735591 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.727216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.16.79.62:46509)
0620 23:15:15.727331 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.727333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.727340 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:15:15.727360 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.727360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.727369 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.727420 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.728150 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.728946 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.730033 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.730320 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.730375 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.732250 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.732251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.732251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.732258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.734155 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.734614 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.734615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.734616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.734666 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.734857 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.734858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.734858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.734977 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.735071 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.735432 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.735502 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:15:15.738106 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.727467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.16.79.61:35349)
0620 23:15:15.727503 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.727504 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.727512 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:15:15.727531 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.727532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.727540 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.727600 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.728207 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.728922 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.730039 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.730320 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.730360 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.732250 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.732251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.732251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.732258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.736925 (+ 4667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.737315 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.737316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.737316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.737367 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.737540 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.737540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.737541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.737672 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.737754 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.737945 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.738018 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:15:15.742599 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.727334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.16.79.60:46753)
0620 23:15:15.727369 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.727371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.727378 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:15:15.727396 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.727397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.727406 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.727450 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.736766 (+ 9316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.737524 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.738513 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.738797 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.738829 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.740030 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.740031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.740032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.740038 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.741475 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.741829 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.741830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.741830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.741879 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.742081 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.742082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.742082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.742191 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.742303 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.742411 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.742483 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:15:15.757911 30152 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } attempt: 3
Jun 20 23:15:15 dist-test-slave-pswb krb5kdc[29996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.61@KRBTEST.COM for kudu/127.16.79.60@KRBTEST.COM
I20260620 23:15:15.765223 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.758563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.16.79.60:46753)
0620 23:15:15.758599 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:15.758601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:15.758650 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:15:15.758758 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:15.758759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:15.758769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:15.758852 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:15.759181 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.759656 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.760463 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.760753 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.760812 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.762710 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:15.762711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:15.762712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:15.762719 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:15.764065 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.764426 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.764427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.764428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.764508 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:15.764730 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:15.764731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:15.764731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:15.764834 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:15.764910 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:15.765058 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 23:15:15.765128 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:15:15.765298 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:15.758515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:46753 (local address 127.0.0.1:40258)
0620 23:15:15.758580 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:15.758593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:15.758667 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:15.758857 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:15.758858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:15.759056 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:15.759098 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.759100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.759740 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.759741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.760372 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:15.760374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:15.760864 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:15.760865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:15.760923 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:15.761407 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:15.761413 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:15.762616 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:15.764176 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.764177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.764177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.764341 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.764583 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:15.764584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:15.764584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:15.764652 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:15.764904 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:15.764905 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:15.764946 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:15:15.765036 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:15:15.765128 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:15:15.767182 30152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a: Bootstrap starting.
I20260620 23:15:15.768332 30152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:15.768750 30152 log.cc:824] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:15.770119 30152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a: No bootstrap required, opened a new log
I20260620 23:15:15.771761 30152 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } }
I20260620 23:15:15.771919 30152 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:15.771987 30152 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7d3e95ec5a74089b3a31d528955547a, State: Initialized, Role: FOLLOWER
I20260620 23:15:15.772140 30152 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } }
I20260620 23:15:15.772627 30152 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:15:15.774848 30280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:15:15.774963 30280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:15:15.777397 30084 master_runner.cc:263] No leader master found from master c7d3e95ec5a74089b3a31d528955547a
I20260620 23:15:15.777496 30084 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:15:15.772570 30269 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } } }
I20260620 23:15:15.778357 30269 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:15:15.857163 30030 tablet.cc:2369] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:15:15.936182 30180 tablet.cc:2369] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:15.944897 30234 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:15.944999 30234 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } }
I20260620 23:15:15.945292 30234 leader_election.cc:288] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7d3e95ec5a74089b3a31d528955547a (127.16.79.61:35349), 7f1ae3720cc643a1b6fbc55547863d5c (127.16.79.60:46753)
I20260620 23:15:15.945577 30201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" is_pre_election: true
I20260620 23:15:15.945744 30201 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac97ef9234734f03a835cc5fe61ac2ee in term 0.
I20260620 23:15:15.945717 30126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7d3e95ec5a74089b3a31d528955547a" is_pre_election: true
I20260620 23:15:15.945844 30126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac97ef9234734f03a835cc5fe61ac2ee in term 0.
I20260620 23:15:15.945953 30025 leader_election.cc:302] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f1ae3720cc643a1b6fbc55547863d5c, ac97ef9234734f03a835cc5fe61ac2ee; no voters:
I20260620 23:15:15.946154 30234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:15:15.946241 30234 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:15:15.946274 30234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:15.947009 30234 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } }
I20260620 23:15:15.947150 30234 leader_election.cc:288] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [CANDIDATE]: Term 1 election: Requested vote from peers c7d3e95ec5a74089b3a31d528955547a (127.16.79.61:35349), 7f1ae3720cc643a1b6fbc55547863d5c (127.16.79.60:46753)
I20260620 23:15:15.947350 30126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7d3e95ec5a74089b3a31d528955547a"
I20260620 23:15:15.947345 30201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1ae3720cc643a1b6fbc55547863d5c"
I20260620 23:15:15.947428 30201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:15.947428 30126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:15.948225 30126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac97ef9234734f03a835cc5fe61ac2ee in term 1.
I20260620 23:15:15.948225 30201 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac97ef9234734f03a835cc5fe61ac2ee in term 1.
I20260620 23:15:15.948421 30025 leader_election.cc:302] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f1ae3720cc643a1b6fbc55547863d5c, ac97ef9234734f03a835cc5fe61ac2ee; no voters:
I20260620 23:15:15.948540 30234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:15.948730 30234 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [term 1 LEADER]: Becoming Leader. State: Replica: ac97ef9234734f03a835cc5fe61ac2ee, State: Running, Role: LEADER
I20260620 23:15:15.948825 30234 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } }
I20260620 23:15:15.949250 30282 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac97ef9234734f03a835cc5fe61ac2ee. Latest consensus state: current_term: 1 leader_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } } }
I20260620 23:15:15.949350 30282 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:15.949522 30284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:15.949721 30284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:15.951277 30201 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [term 1 FOLLOWER]: Refusing update from remote peer ac97ef9234734f03a835cc5fe61ac2ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:15.951325 30126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [term 1 FOLLOWER]: Refusing update from remote peer ac97ef9234734f03a835cc5fe61ac2ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:15.951627 30234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:15.951937 30234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:15.952809 30254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac97ef9234734f03a835cc5fe61ac2ee. Latest consensus state: current_term: 1 leader_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } } }
I20260620 23:15:15.952900 30254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:15:15.953883 30269 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac97ef9234734f03a835cc5fe61ac2ee. Latest consensus state: current_term: 1 leader_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } } }
I20260620 23:15:15.953935 30234 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } } }
I20260620 23:15:15.953989 30269 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:15:15.953998 30234 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:15.954329 30282 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } } }
I20260620 23:15:15.954460 30282 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:15.954912 30254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } } }
I20260620 23:15:15.954998 30254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:15:15.955026 30269 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac97ef9234734f03a835cc5fe61ac2ee" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 46509 } } peers { permanent_uuid: "c7d3e95ec5a74089b3a31d528955547a" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 35349 } } peers { permanent_uuid: "7f1ae3720cc643a1b6fbc55547863d5c" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 46753 } } }
I20260620 23:15:15.955134 30269 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:15:15.955157 30288 mvcc.cc:205] Tried to move back new op lower bound from 7299061006133489664 to 7299061006127009792. Current Snapshot: MvccSnapshot[applied={T|T < 7299061006133489664}]
I20260620 23:15:15.955258 30284 catalog_manager.cc:1347] Generated new cluster ID: 12e56045a8c2478b9e0e21029d40ffa7
I20260620 23:15:15.955313 30284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:15.979070 30284 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:15:15.979681 30284 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:15.998831 30284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac97ef9234734f03a835cc5fe61ac2ee: Generated new TSK 0
I20260620 23:15:15.999029 30284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:15:16.732805 30266 catalog_manager.cc:1259] Loaded cluster ID: 12e56045a8c2478b9e0e21029d40ffa7
I20260620 23:15:16.732915 30266 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c: loading cluster ID for follower catalog manager: success
I20260620 23:15:16.733855 30266 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c: acquiring CA information for follower catalog manager: success
I20260620 23:15:16.734165 30266 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f1ae3720cc643a1b6fbc55547863d5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:15:16.775466 30280 catalog_manager.cc:1259] Loaded cluster ID: 12e56045a8c2478b9e0e21029d40ffa7
I20260620 23:15:16.775547 30280 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a: loading cluster ID for follower catalog manager: success
I20260620 23:15:16.776386 30280 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a: acquiring CA information for follower catalog manager: success
I20260620 23:15:16.776741 30280 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7d3e95ec5a74089b3a31d528955547a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:15:17.070230 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 30009
I20260620 23:15:17.076061 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 30084
I20260620 23:15:17.082079 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 30159
2026-06-20T23:15:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3029 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:15:17 dist-test-slave-pswb krb5kdc[30300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:15:17 dist-test-slave-pswb krb5kdc[30300](info): set up 2 sockets
Jun 20 23:15:17 dist-test-slave-pswb krb5kdc[30300](info): commencing operation
krb5kdc: starting...
W20260620 23:15:18.144956 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:15:18 dist-test-slave-pswb krb5kdc[30300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:15:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:15:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:15:18.267791 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:37185
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:37185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:18.347848 30313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:18.348037 30313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:18.348058 30313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:18.349114 30313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:18.349150 30313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:18.349164 30313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:18.349177 30313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:18.349190 30313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:18.350544 30313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:37185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:37185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:18.350716 30313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:18.350871 30313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:18.355449 30318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:18.355454 30321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:18.355593 30319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:18.355813 30313 server_base.cc:1034] running on GCE node
I20260620 23:15:18.356030 30313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:18.356251 30313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:18.357409 30313 hybrid_clock.cc:648] HybridClock initialized: now 1781997318357404 us; error 37 us; skew 500 ppm
Jun 20 23:15:18 dist-test-slave-pswb krb5kdc[30300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997318, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:18.360116 30313 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:15:18.360605 30313 webserver.cc:460] Webserver started at http://127.16.79.62:41095/ using document root <none> and password file <none>
I20260620 23:15:18.360859 30313 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:18.360929 30313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:18.361084 30313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:18.362097 30313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "bfe505b0c1e646b2aaee6fbe47c340d3"
format_stamp: "Formatted at 2026-06-20 23:15:18 on dist-test-slave-pswb"
server_key: "0e4f2d07b4c80a52b6a4dc460147b7e3"
server_key_iv: "440e1f4600c9ccb96ae8ad4a8e8d37dd"
server_key_version: "encryptionkey@0"
I20260620 23:15:18.362452 30313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "bfe505b0c1e646b2aaee6fbe47c340d3"
format_stamp: "Formatted at 2026-06-20 23:15:18 on dist-test-slave-pswb"
server_key: "0e4f2d07b4c80a52b6a4dc460147b7e3"
server_key_iv: "440e1f4600c9ccb96ae8ad4a8e8d37dd"
server_key_version: "encryptionkey@0"
I20260620 23:15:18.363765 30313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:15:18.364514 30328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:18.364765 30313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:18.364856 30313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "bfe505b0c1e646b2aaee6fbe47c340d3"
format_stamp: "Formatted at 2026-06-20 23:15:18 on dist-test-slave-pswb"
server_key: "0e4f2d07b4c80a52b6a4dc460147b7e3"
server_key_iv: "440e1f4600c9ccb96ae8ad4a8e8d37dd"
server_key_version: "encryptionkey@0"
I20260620 23:15:18.364943 30313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:18.374147 30313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:18.376114 30313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:18.376271 30313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:18.380281 30313 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:37185
I20260620 23:15:18.380316 30380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:37185 every 8 connection(s)
I20260620 23:15:18.380638 30313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:18.381523 30381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:18.382530 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 30313
I20260620 23:15:18.382653 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:15:18.382843 16700 external_mini_cluster.cc:1428] Setting key 2465072d9ee220789c8ef66c2b6d9dc9
I20260620 23:15:18.384258 30381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3: Bootstrap starting.
I20260620 23:15:18.385015 30381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:18.385325 30381 log.cc:824] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:18.385967 30381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3: No bootstrap required, opened a new log
I20260620 23:15:18.388238 30381 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfe505b0c1e646b2aaee6fbe47c340d3" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 37185 } }
I20260620 23:15:18.388410 30381 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:18.388448 30381 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe505b0c1e646b2aaee6fbe47c340d3, State: Initialized, Role: FOLLOWER
I20260620 23:15:18.388573 30381 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfe505b0c1e646b2aaee6fbe47c340d3" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 37185 } }
I20260620 23:15:18.388675 30381 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:18.388734 30381 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:18.388787 30381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:18.389467 30381 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfe505b0c1e646b2aaee6fbe47c340d3" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 37185 } }
I20260620 23:15:18.389552 30381 leader_election.cc:302] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfe505b0c1e646b2aaee6fbe47c340d3; no voters:
I20260620 23:15:18.389775 30381 leader_election.cc:288] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:15:18.389803 30386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [term 1 FOLLOWER]: Leader election won for term 1
Jun 20 23:15:18 dist-test-slave-pswb krb5kdc[30300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:18.389916 30386 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [term 1 LEADER]: Becoming Leader. State: Replica: bfe505b0c1e646b2aaee6fbe47c340d3, State: Running, Role: LEADER
I20260620 23:15:18.389986 30386 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfe505b0c1e646b2aaee6fbe47c340d3" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 37185 } }
I20260620 23:15:18.390030 30381 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:18.390691 30386 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfe505b0c1e646b2aaee6fbe47c340d3. Latest consensus state: current_term: 1 leader_uuid: "bfe505b0c1e646b2aaee6fbe47c340d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfe505b0c1e646b2aaee6fbe47c340d3" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 37185 } } }
I20260620 23:15:18.390692 30387 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfe505b0c1e646b2aaee6fbe47c340d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfe505b0c1e646b2aaee6fbe47c340d3" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 37185 } } }
I20260620 23:15:18.390802 30386 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:18.390805 30387 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3 [sys.catalog]: This master's current role is: LEADER
W20260620 23:15:18.391852 30401 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:15:18.391912 30401 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:15:18.392560 30395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:18.392772 30395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:18.394101 30384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:18.383311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.16.79.62:37185)
0620 23:15:18.383485 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:18.383492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:18.383515 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:15:18.384046 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:18.384052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:18.384094 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:18.384205 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:18.385033 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:18.385980 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:18.387026 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:18.387420 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:18.387498 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:18.390339 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:18.390342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:18.390343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:18.390356 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:18.392470 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:18.392900 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:18.392903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:18.392904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:18.393005 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:18.393211 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:18.393212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:18.393212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:18.393500 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:18.393650 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:18.393785 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 23:15:18.393916 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260620 23:15:18.394590 30395 catalog_manager.cc:1347] Generated new cluster ID: a92c8d10a7bf49f3a980b35b49a5c7e3
I20260620 23:15:18.394641 30395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:18.401261 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:15:18.403148 30395 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:15:18.403658 30395 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:18.414845 30395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfe505b0c1e646b2aaee6fbe47c340d3: Generated new TSK 0
I20260620 23:15:18.415016 30395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:15:18.455297 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 30313
2026-06-20T23:15:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1370 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:15:18 dist-test-slave-pswb krb5kdc[30408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:15:18 dist-test-slave-pswb krb5kdc[30408](info): set up 2 sockets
Jun 20 23:15:18 dist-test-slave-pswb krb5kdc[30408](info): commencing operation
krb5kdc: starting...
W20260620 23:15:19.512921 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:15:19 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:15:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:15:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:15:19.635241 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:40353
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:40353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:19.717490 30421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:19.717679 30421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:19.717700 30421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:19.718816 30421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:19.718856 30421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:19.718871 30421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:19.718883 30421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:19.718895 30421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:19.720351 30421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:40353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:40353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:19.720584 30421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:19.720844 30421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:19.726233 30427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:19.726341 30426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:19.726380 30429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:19.726447 30421 server_base.cc:1034] running on GCE node
I20260620 23:15:19.726739 30421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:19.727010 30421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:19.728181 30421 hybrid_clock.cc:648] HybridClock initialized: now 1781997319728181 us; error 40 us; skew 500 ppm
Jun 20 23:15:19 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997319, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:19.730928 30421 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:15:19.731443 30421 webserver.cc:460] Webserver started at http://127.16.79.62:41845/ using document root <none> and password file <none>
I20260620 23:15:19.731698 30421 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:19.731771 30421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:19.731922 30421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:19.733014 30421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "ac3dc4731ae04cada98b786e2659e5bb"
format_stamp: "Formatted at 2026-06-20 23:15:19 on dist-test-slave-pswb"
server_key: "7582e05850f649ea71b4e1eaad87d552"
server_key_iv: "64009d54fc48b05a8496b36b774b9b48"
server_key_version: "encryptionkey@0"
I20260620 23:15:19.733354 30421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "ac3dc4731ae04cada98b786e2659e5bb"
format_stamp: "Formatted at 2026-06-20 23:15:19 on dist-test-slave-pswb"
server_key: "7582e05850f649ea71b4e1eaad87d552"
server_key_iv: "64009d54fc48b05a8496b36b774b9b48"
server_key_version: "encryptionkey@0"
I20260620 23:15:19.734760 30421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:19.735865 30436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:19.736155 30421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:19.736280 30421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "ac3dc4731ae04cada98b786e2659e5bb"
format_stamp: "Formatted at 2026-06-20 23:15:19 on dist-test-slave-pswb"
server_key: "7582e05850f649ea71b4e1eaad87d552"
server_key_iv: "64009d54fc48b05a8496b36b774b9b48"
server_key_version: "encryptionkey@0"
I20260620 23:15:19.736357 30421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:19.750644 30421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:19.752904 30421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:19.753058 30421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:19.756484 30421 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:40353
I20260620 23:15:19.756536 30488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:40353 every 8 connection(s)
I20260620 23:15:19.756913 30421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:19.758200 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:19.760409 30489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Bootstrap starting.
I20260620 23:15:19.760648 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 30421
I20260620 23:15:19.760783 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:15:19.760983 16700 external_mini_cluster.cc:1428] Setting key 5fa8ca727adc63c05b9ecbc087adff78
I20260620 23:15:19.761132 30489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:19.761480 30489 log.cc:824] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:19.762243 30489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: No bootstrap required, opened a new log
I20260620 23:15:19.764472 30489 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } }
I20260620 23:15:19.764644 30489 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:19.764678 30489 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3dc4731ae04cada98b786e2659e5bb, State: Initialized, Role: FOLLOWER
I20260620 23:15:19.764824 30489 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } }
I20260620 23:15:19.764905 30489 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:19.764932 30489 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:19.764967 30489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:19.765555 30489 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } }
I20260620 23:15:19.765638 30489 leader_election.cc:302] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3dc4731ae04cada98b786e2659e5bb; no voters:
I20260620 23:15:19.765815 30489 leader_election.cc:288] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:15:19.765841 30494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:19.766005 30494 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 1 LEADER]: Becoming Leader. State: Replica: ac3dc4731ae04cada98b786e2659e5bb, State: Running, Role: LEADER
I20260620 23:15:19.766072 30489 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:19.766103 30494 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } }
I20260620 23:15:19.766403 30494 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac3dc4731ae04cada98b786e2659e5bb. Latest consensus state: current_term: 1 leader_uuid: "ac3dc4731ae04cada98b786e2659e5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } } }
I20260620 23:15:19.766492 30494 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:19.766808 30500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:19.767273 30500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:19.767926 30496 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac3dc4731ae04cada98b786e2659e5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } } }
Jun 20 23:15:19 dist-test-slave-pswb krb5kdc[30408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:19.768254 30496 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:19.769484 30500 catalog_manager.cc:1347] Generated new cluster ID: 330dc88e8ee0446f933b6e832b293b3f
I20260620 23:15:19.769537 30500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:19.772548 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:19.761477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.16.79.62:40353)
0620 23:15:19.761749 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:19.761757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:19.761787 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:15:19.762285 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:19.762291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:19.762359 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:19.762494 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:19.763263 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:19.764138 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:19.765209 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:19.765594 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:19.765734 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:19.768774 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:19.768778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:19.768778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:19.768790 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:19.770970 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:19.771337 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:19.771340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:19.771340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:19.771443 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:19.771644 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:19.771645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:19.771646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:19.771940 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:19.772113 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:19.772211 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 23:15:19.772342 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
I20260620 23:15:19.784512 30500 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:15:19.785252 30500 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:19.789928 30500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Generated new TSK 0
I20260620 23:15:19.790102 30500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:15:19.821521 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:40353
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:19.923326 30515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:19.923728 30515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:19.923769 30515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:19.924988 30515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:19.925050 30515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:19.925099 30515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:15:19.926582 30515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:19.926826 30515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:19.927044 30515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:19.931954 30523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:19.932133 30515 server_base.cc:1034] running on GCE node
W20260620 23:15:19.931931 30521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:19.932438 30520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:19.932804 30515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:19.933069 30515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:19.934267 30515 hybrid_clock.cc:648] HybridClock initialized: now 1781997319934259 us; error 38 us; skew 500 ppm
Jun 20 23:15:19 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997319, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:19.936873 30515 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:15:19.937314 30515 webserver.cc:460] Webserver started at http://127.16.79.1:44513/ using document root <none> and password file <none>
I20260620 23:15:19.937553 30515 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:19.937621 30515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:19.937755 30515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:19.938796 30515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "11482a723bfa4625a5c84d8eb153c679"
format_stamp: "Formatted at 2026-06-20 23:15:19 on dist-test-slave-pswb"
server_key: "ea46306dd56975fa7b35557ff2f90d16"
server_key_iv: "0f645c1ebd12a76b94d67aeb3ff38084"
server_key_version: "encryptionkey@0"
I20260620 23:15:19.939162 30515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "11482a723bfa4625a5c84d8eb153c679"
format_stamp: "Formatted at 2026-06-20 23:15:19 on dist-test-slave-pswb"
server_key: "ea46306dd56975fa7b35557ff2f90d16"
server_key_iv: "0f645c1ebd12a76b94d67aeb3ff38084"
server_key_version: "encryptionkey@0"
I20260620 23:15:19.940724 30515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:15:19.941838 30530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:19.942138 30515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:19.942251 30515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "11482a723bfa4625a5c84d8eb153c679"
format_stamp: "Formatted at 2026-06-20 23:15:19 on dist-test-slave-pswb"
server_key: "ea46306dd56975fa7b35557ff2f90d16"
server_key_iv: "0f645c1ebd12a76b94d67aeb3ff38084"
server_key_version: "encryptionkey@0"
I20260620 23:15:19.942340 30515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:19.956808 30515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:19.959139 30515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:19.959311 30515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:19.959558 30515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:19.959867 30515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:19.959920 30515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:19.959964 30515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:19.959990 30515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:19.965914 30515 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:33585
I20260620 23:15:19.966050 30643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:33585 every 8 connection(s)
I20260620 23:15:19.966300 30515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:15:19.967299 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 30515
I20260620 23:15:19.967435 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:15:19.967710 16700 external_mini_cluster.cc:1428] Setting key c06c1a47ff435fd0511f7f55d8d3273c
Jun 20 23:15:19 dist-test-slave-pswb krb5kdc[30408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997319, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
I20260620 23:15:19.977169 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:19.967470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:37617 (local address 127.16.79.62:40353)
0620 23:15:19.967514 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:19.967517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:19.967809 (+ 292us) server_negotiation.cc:407] Connection header received
0620 23:15:19.968409 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:19.968410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:19.968425 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:19.968476 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:19.969461 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:19.969978 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:19.970953 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:19.971172 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:19.971257 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:19.974018 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:19.974021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:19.974021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:19.974031 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:19.975674 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:19.976132 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:19.976134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:19.976134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:19.976202 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:19.976473 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:19.976474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:19.976475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:19.976589 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:19.976706 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:19.976975 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 23:15:19.977061 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:15:19.977388 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:19.967460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:37617)
0620 23:15:19.967677 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:19.967715 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:19.968294 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:19.968547 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:19.968551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:19.969173 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:19.969362 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:19.969366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:19.970076 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:19.970077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:19.970851 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:19.970853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:19.971268 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:19.971269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:19.971385 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:19.971965 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:19.971973 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:19.973888 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:19.975802 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:19.975808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:19.975810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:19.976038 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:19.976281 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:19.976281 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:19.976282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:19.976407 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:19.976686 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:19.976713 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:19.976847 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 23:15:19.976955 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:15:19.977159 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":135,"threads_started":1}
I20260620 23:15:19.977870 30644 heartbeater.cc:347] Connected to a master server at 127.16.79.62:40353
I20260620 23:15:19.977957 30644 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:19.978175 30644 heartbeater.cc:511] Master 127.16.79.62:40353 requested a full tablet report, sending...
I20260620 23:15:19.978809 30453 ts_manager.cc:194] Registered new tserver with Master: 11482a723bfa4625a5c84d8eb153c679 (127.16.79.1:33585)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
I20260620 23:15:19.979514 30453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:37617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:15:19.995647 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:40353
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:20.082655 30649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:20.082867 30649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:20.082893 30649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:20.084010 30649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:20.084050 30649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:20.084077 30649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:15:20.085492 30649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:20.085685 30649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:20.085855 30649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:20.090999 30655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:20.091073 30654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:20.091289 30657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:20.091212 30649 server_base.cc:1034] running on GCE node
I20260620 23:15:20.091536 30649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:20.091748 30649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:20.093046 30649 hybrid_clock.cc:648] HybridClock initialized: now 1781997320093005 us; error 39 us; skew 500 ppm
Jun 20 23:15:20 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:20.096390 30649 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:15:20.097072 30649 webserver.cc:460] Webserver started at http://127.16.79.2:40621/ using document root <none> and password file <none>
I20260620 23:15:20.097496 30649 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:20.097579 30649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:20.097741 30649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:20.099063 30649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "6f562397a5d54eac9ac484eaed971eb1"
format_stamp: "Formatted at 2026-06-20 23:15:20 on dist-test-slave-pswb"
server_key: "fba84ba67fa8ad300657e0f4ea858d70"
server_key_iv: "75ecb2da8b18884213f738407126b082"
server_key_version: "encryptionkey@0"
I20260620 23:15:20.099495 30649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "6f562397a5d54eac9ac484eaed971eb1"
format_stamp: "Formatted at 2026-06-20 23:15:20 on dist-test-slave-pswb"
server_key: "fba84ba67fa8ad300657e0f4ea858d70"
server_key_iv: "75ecb2da8b18884213f738407126b082"
server_key_version: "encryptionkey@0"
I20260620 23:15:20.101250 30649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 23:15:20.102289 30664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.102655 30649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:20.102764 30649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "6f562397a5d54eac9ac484eaed971eb1"
format_stamp: "Formatted at 2026-06-20 23:15:20 on dist-test-slave-pswb"
server_key: "fba84ba67fa8ad300657e0f4ea858d70"
server_key_iv: "75ecb2da8b18884213f738407126b082"
server_key_version: "encryptionkey@0"
I20260620 23:15:20.102849 30649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:20.114133 30649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:20.116474 30649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:20.116631 30649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:20.116869 30649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:20.117180 30649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:20.117234 30649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.117274 30649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:20.117302 30649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.122783 30649 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:34525
I20260620 23:15:20.122855 30777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:34525 every 8 connection(s)
I20260620 23:15:20.123155 30649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:15:20 dist-test-slave-pswb krb5kdc[30408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:20.130601 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 30649
I20260620 23:15:20.130731 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:15:20.130913 16700 external_mini_cluster.cc:1428] Setting key d182618c5582871a2c7dcadec0afa75a
I20260620 23:15:20.133134 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.124122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:43823 (local address 127.16.79.62:40353)
0620 23:15:20.124206 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.124208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.124347 (+ 139us) server_negotiation.cc:407] Connection header received
0620 23:15:20.124909 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.124910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.124921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.124983 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:20.126001 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.126495 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.127455 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.127646 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.127708 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.130227 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:20.130228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:20.130228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:20.130234 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:20.131777 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.132194 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.132195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.132196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.132254 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.132496 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.132497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.132497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.132604 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:20.132677 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.132970 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.133044 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:15:20.133311 30780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.124069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:43823)
0620 23:15:20.124228 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.124268 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:20.124794 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:20.125047 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:20.125051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:20.125605 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:20.125787 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.125791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.126604 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.126605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.127363 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.127365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.127737 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.127738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.127841 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.128516 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:20.128525 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:20.130124 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:20.131878 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:20.131885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:20.131887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:20.132094 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:20.132324 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:20.132324 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:20.132325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:20.132424 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:20.132682 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:20.132688 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:20.132843 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 23:15:20.132947 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:15:20.133096 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260620 23:15:20.133951 30778 heartbeater.cc:347] Connected to a master server at 127.16.79.62:40353
I20260620 23:15:20.134054 30778 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:20.134230 30778 heartbeater.cc:511] Master 127.16.79.62:40353 requested a full tablet report, sending...
I20260620 23:15:20.134927 30453 ts_manager.cc:194] Registered new tserver with Master: 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525)
I20260620 23:15:20.135651 30453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:43823
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:15:20.160189 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:40353
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:20.239305 30783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:20.239678 30783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:20.239818 30783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:20.242017 30783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:20.242093 30783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:20.242131 30783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:15:20.244782 30783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:20.245021 30783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:20.245258 30783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:20.250327 30791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:20.250351 30783 server_base.cc:1034] running on GCE node
W20260620 23:15:20.250355 30788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:20.250351 30789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:20.250797 30783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:20.251024 30783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:20.252194 30783 hybrid_clock.cc:648] HybridClock initialized: now 1781997320252187 us; error 33 us; skew 500 ppm
Jun 20 23:15:20 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:20.254653 30783 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:15:20.255012 30783 webserver.cc:460] Webserver started at http://127.16.79.3:46557/ using document root <none> and password file <none>
I20260620 23:15:20.255237 30783 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:20.255287 30783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:20.255403 30783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:20.256398 30783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "cce78ca247f742c18dfad6ea424477ad"
format_stamp: "Formatted at 2026-06-20 23:15:20 on dist-test-slave-pswb"
server_key: "699176874c5f29785f596fa795661173"
server_key_iv: "cff7cb8c1a59a7f9fdfa6f0e0d10518a"
server_key_version: "encryptionkey@0"
I20260620 23:15:20.256779 30783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "cce78ca247f742c18dfad6ea424477ad"
format_stamp: "Formatted at 2026-06-20 23:15:20 on dist-test-slave-pswb"
server_key: "699176874c5f29785f596fa795661173"
server_key_iv: "cff7cb8c1a59a7f9fdfa6f0e0d10518a"
server_key_version: "encryptionkey@0"
I20260620 23:15:20.258066 30783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:20.258932 30798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.259127 30783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:20.259217 30783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "cce78ca247f742c18dfad6ea424477ad"
format_stamp: "Formatted at 2026-06-20 23:15:20 on dist-test-slave-pswb"
server_key: "699176874c5f29785f596fa795661173"
server_key_iv: "cff7cb8c1a59a7f9fdfa6f0e0d10518a"
server_key_version: "encryptionkey@0"
I20260620 23:15:20.259294 30783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:20.280275 30783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:20.282646 30783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:20.282804 30783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:20.283022 30783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:20.283315 30783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:20.283365 30783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.283406 30783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:20.283463 30783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.289547 30911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:36389 every 8 connection(s)
I20260620 23:15:20.289599 30783 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:36389
I20260620 23:15:20.290131 30783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:15:20.295449 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 30783
I20260620 23:15:20.295647 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:15:20.295935 16700 external_mini_cluster.cc:1428] Setting key 43bb5cad667503527573458dbf4c3b59
Jun 20 23:15:20 dist-test-slave-pswb krb5kdc[30408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:20.300729 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.290839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:49893 (local address 127.16.79.62:40353)
0620 23:15:20.290940 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.290942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.291164 (+ 222us) server_negotiation.cc:407] Connection header received
0620 23:15:20.291618 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.291618 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.291628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.291686 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:20.292591 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.293297 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.294368 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.294614 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.294714 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.297540 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:20.297542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:20.297543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:20.297556 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:20.299314 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.299725 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.299726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.299726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.299784 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.300066 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.300068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.300068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.300225 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:20.300297 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.300534 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.300611 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:15:20.300945 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.290844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:49893)
0620 23:15:20.291050 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.291082 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:20.291540 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:20.291743 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:20.291746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:20.292294 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:20.292479 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.292484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.293396 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.293397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.294227 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.294230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.294740 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.294741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.294848 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.295807 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:20.295821 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:20.297401 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:20.299412 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:20.299418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:20.299419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:20.299644 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:20.299849 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:20.299850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:20.299850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:20.299965 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:20.300301 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:20.300306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:20.300423 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 23:15:20.300510 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:15:20.300625 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":147,"threads_started":1}
I20260620 23:15:20.301671 30912 heartbeater.cc:347] Connected to a master server at 127.16.79.62:40353
I20260620 23:15:20.301755 30912 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:20.301879 30912 heartbeater.cc:511] Master 127.16.79.62:40353 requested a full tablet report, sending...
I20260620 23:15:20.302258 30453 ts_manager.cc:194] Registered new tserver with Master: cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389)
I20260620 23:15:20.302600 30453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:49893
I20260620 23:15:20.308163 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:15:20.315945 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.309540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.16.79.62:40353)
0620 23:15:20.309589 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.309591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.309738 (+ 147us) server_negotiation.cc:407] Connection header received
0620 23:15:20.309814 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.309815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.309827 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.309894 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:20.310429 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.310900 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.311624 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.311811 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.312013 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.313388 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:20.313390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:20.313390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:20.313398 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:20.314793 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.315150 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.315151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.315152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.315209 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.315406 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.315407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.315407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.315550 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:20.315680 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.315766 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.315845 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:15:20.317627 30453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:15:20.319183 30453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:15:20.325675 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.321344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:33585 (local address 127.0.0.1:38824)
0620 23:15:20.321712 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.321721 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:20.321758 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:20.321986 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:20.321989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:20.321991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:20.322054 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.322056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.323617 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.323618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.324670 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.324672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.325380 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.325381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.325458 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.325472 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:15:20.325552 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:15:20.325603 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":79,"threads_started":1}
I20260620 23:15:20.325709 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.321606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.16.79.1:33585)
0620 23:15:20.321793 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.321797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.321805 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:15:20.321835 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.321837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.321848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.321958 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:20.322154 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.323441 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.324811 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.325306 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.325367 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.325404 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.325589 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.325630 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260620 23:15:20.326387 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.321336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34525 (local address 127.0.0.1:35692)
0620 23:15:20.321492 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.321538 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:20.321602 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:20.322300 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:20.322301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:20.322303 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:20.322347 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.322348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.323893 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.323895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.325105 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.325108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.326126 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.326127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.326204 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.326208 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:15:20.326260 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 23:15:20.326309 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260620 23:15:20.326469 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.321336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:36389 (local address 127.0.0.1:54246)
0620 23:15:20.321504 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.321537 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:20.321600 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:20.322096 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:20.322097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:20.322099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:20.322142 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.322144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.323119 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.323120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.324265 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.324268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.326240 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.326241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.326309 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.326313 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:15:20.326364 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 23:15:20.326404 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260620 23:15:20.326697 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.321598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.16.79.2:34525)
0620 23:15:20.322085 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.322089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.322098 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:15:20.322135 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.322138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.322152 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.322234 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:20.322421 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.323600 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.325213 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.325938 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.326005 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.326072 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.326577 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.326622 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":83,"threads_started":1}
I20260620 23:15:20.326711 30578 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee87760bd2e48e29fcf81aba3c996ef (DEFAULT_TABLE table=test-table [id=bbdf6b0d774148ba83f6298dfe533d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:20.326957 30928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.321773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.16.79.3:36389)
0620 23:15:20.321929 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.321932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.321941 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:15:20.321975 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.321977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.321992 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.322218 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:20.322277 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.323023 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.325433 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.326069 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.326144 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.326193 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.326393 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.326873 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260620 23:15:20.327077 30578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee87760bd2e48e29fcf81aba3c996ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:20.327553 30846 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee87760bd2e48e29fcf81aba3c996ef (DEFAULT_TABLE table=test-table [id=bbdf6b0d774148ba83f6298dfe533d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:20.327842 30846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee87760bd2e48e29fcf81aba3c996ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:20.330493 30929 tablet_bootstrap.cc:492] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Bootstrap starting.
I20260620 23:15:20.330996 30712 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee87760bd2e48e29fcf81aba3c996ef (DEFAULT_TABLE table=test-table [id=bbdf6b0d774148ba83f6298dfe533d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:20.331215 30929 tablet_bootstrap.cc:654] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:20.331215 30930 tablet_bootstrap.cc:492] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad: Bootstrap starting.
I20260620 23:15:20.331313 30712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee87760bd2e48e29fcf81aba3c996ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:20.331816 30929 log.cc:824] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:20.331851 30930 tablet_bootstrap.cc:654] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:20.332404 30930 log.cc:824] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:20.332614 30929 tablet_bootstrap.cc:492] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: No bootstrap required, opened a new log
I20260620 23:15:20.332727 30929 ts_tablet_manager.cc:1397] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:15:20.333125 30930 tablet_bootstrap.cc:492] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad: No bootstrap required, opened a new log
I20260620 23:15:20.333207 30930 ts_tablet_manager.cc:1397] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:15:20.334139 30933 tablet_bootstrap.cc:492] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1: Bootstrap starting.
I20260620 23:15:20.334697 30933 tablet_bootstrap.cc:654] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:20.334688 30929 raft_consensus.cc:348] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:20.334854 30929 raft_consensus.cc:374] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:20.334923 30929 raft_consensus.cc:729] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11482a723bfa4625a5c84d8eb153c679, State: Initialized, Role: FOLLOWER
I20260620 23:15:20.335065 30929 consensus_queue.cc:260] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:20.335078 30930 raft_consensus.cc:348] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:20.335237 30933 log.cc:824] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:20.335273 30930 raft_consensus.cc:374] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:20.335319 30930 raft_consensus.cc:729] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce78ca247f742c18dfad6ea424477ad, State: Initialized, Role: FOLLOWER
I20260620 23:15:20.335340 30929 ts_tablet_manager.cc:1428] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:15:20.335484 30644 heartbeater.cc:503] Master 127.16.79.62:40353 was elected leader, sending a full tablet report...
I20260620 23:15:20.335480 30930 consensus_queue.cc:260] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:20.335748 30930 ts_tablet_manager.cc:1428] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:15:20.335881 30912 heartbeater.cc:503] Master 127.16.79.62:40353 was elected leader, sending a full tablet report...
I20260620 23:15:20.336416 30933 tablet_bootstrap.cc:492] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1: No bootstrap required, opened a new log
I20260620 23:15:20.336478 30933 ts_tablet_manager.cc:1397] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:15:20.338146 30933 raft_consensus.cc:348] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:20.338258 30933 raft_consensus.cc:374] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:20.338279 30933 raft_consensus.cc:729] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f562397a5d54eac9ac484eaed971eb1, State: Initialized, Role: FOLLOWER
I20260620 23:15:20.338371 30933 consensus_queue.cc:260] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:20.338591 30933 ts_tablet_manager.cc:1428] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:15:20.338649 30778 heartbeater.cc:503] Master 127.16.79.62:40353 was elected leader, sending a full tablet report...
W20260620 23:15:20.373893 30779 tablet.cc:2369] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:20.394620 30934 raft_consensus.cc:482] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:20.394793 30934 raft_consensus.cc:504] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:20.395200 30934 leader_election.cc:288] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389), 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525)
I20260620 23:15:20.398626 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.395368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34525 (local address 127.16.79.1:43821)
0620 23:15:20.395436 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.395450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:20.395495 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:20.395682 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:20.395683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:20.395685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:20.395764 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.395766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.396562 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.396563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.397686 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.397688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.398432 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.398433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.398495 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.398499 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:15:20.398544 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 23:15:20.398576 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:15:20.398692 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.395437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:43821 (local address 127.16.79.2:34525)
0620 23:15:20.395486 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.395488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.395499 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:15:20.395588 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.395589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.395603 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.395658 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:20.395841 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.396464 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.397809 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.398348 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.398426 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.398438 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.398568 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.398606 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:15:20.398988 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.395376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:36389 (local address 127.16.79.1:33039)
0620 23:15:20.395634 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.395665 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:20.395743 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:20.395968 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:20.395970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:20.395975 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:20.396087 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.396089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.396922 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.396923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.397942 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.397944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.398720 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.398722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.398799 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.398805 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:15:20.398885 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:15:20.398921 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":59,"thread_start_us":125,"threads_started":1}
I20260620 23:15:20.399010 30928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.395465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:33039 (local address 127.16.79.3:36389)
0620 23:15:20.395507 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.395509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.395739 (+ 230us) server_negotiation.cc:407] Connection header received
0620 23:15:20.395817 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.395818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.395831 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.395882 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:20.396208 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.396839 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.398029 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.398608 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.398675 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.398684 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.398902 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.398944 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:15:20.398962 30732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee87760bd2e48e29fcf81aba3c996ef" candidate_uuid: "11482a723bfa4625a5c84d8eb153c679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f562397a5d54eac9ac484eaed971eb1" is_pre_election: true
I20260620 23:15:20.399121 30732 raft_consensus.cc:2454] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11482a723bfa4625a5c84d8eb153c679 in term 0.
I20260620 23:15:20.399387 30532 leader_election.cc:302] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11482a723bfa4625a5c84d8eb153c679, 6f562397a5d54eac9ac484eaed971eb1; no voters:
I20260620 23:15:20.399518 30934 raft_consensus.cc:2781] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:15:20.399467 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee87760bd2e48e29fcf81aba3c996ef" candidate_uuid: "11482a723bfa4625a5c84d8eb153c679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce78ca247f742c18dfad6ea424477ad" is_pre_election: true
I20260620 23:15:20.399571 30934 raft_consensus.cc:482] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:15:20.399595 30934 raft_consensus.cc:3037] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:20.399642 30866 raft_consensus.cc:2454] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11482a723bfa4625a5c84d8eb153c679 in term 0.
I20260620 23:15:20.400348 30934 raft_consensus.cc:504] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:20.400496 30934 leader_election.cc:288] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 1 election: Requested vote from peers cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389), 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525)
I20260620 23:15:20.400674 30732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee87760bd2e48e29fcf81aba3c996ef" candidate_uuid: "11482a723bfa4625a5c84d8eb153c679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f562397a5d54eac9ac484eaed971eb1"
I20260620 23:15:20.400797 30732 raft_consensus.cc:3037] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:20.400859 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee87760bd2e48e29fcf81aba3c996ef" candidate_uuid: "11482a723bfa4625a5c84d8eb153c679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce78ca247f742c18dfad6ea424477ad"
I20260620 23:15:20.401060 30866 raft_consensus.cc:3037] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:20.401571 30732 raft_consensus.cc:2454] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11482a723bfa4625a5c84d8eb153c679 in term 1.
I20260620 23:15:20.401741 30532 leader_election.cc:302] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11482a723bfa4625a5c84d8eb153c679, 6f562397a5d54eac9ac484eaed971eb1; no voters:
I20260620 23:15:20.401847 30934 raft_consensus.cc:2781] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:20.402014 30934 raft_consensus.cc:686] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 1 LEADER]: Becoming Leader. State: Replica: 11482a723bfa4625a5c84d8eb153c679, State: Running, Role: LEADER
I20260620 23:15:20.402110 30934 consensus_queue.cc:237] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:20.402230 30866 raft_consensus.cc:2454] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11482a723bfa4625a5c84d8eb153c679 in term 1.
I20260620 23:15:20.402935 30453 catalog_manager.cc:5495] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11482a723bfa4625a5c84d8eb153c679 (127.16.79.1). New cstate: current_term: 1 leader_uuid: "11482a723bfa4625a5c84d8eb153c679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } health_report { overall_health: HEALTHY } } }
I20260620 23:15:20.422176 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.419947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.16.79.1:33585)
0620 23:15:20.419994 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.419996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.420005 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:15:20.420060 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.420061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.420077 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.420129 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:20.420244 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.420714 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.421476 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.421654 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.421710 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.421841 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:20.421927 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:15:20.421986 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.422073 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.422090 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:15:20.423938 30866 raft_consensus.cc:1261] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad [term 1 FOLLOWER]: Refusing update from remote peer 11482a723bfa4625a5c84d8eb153c679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:20.423951 30732 raft_consensus.cc:1261] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1 [term 1 FOLLOWER]: Refusing update from remote peer 11482a723bfa4625a5c84d8eb153c679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:20.424515 30934 consensus_queue.cc:1035] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:20.424748 30939 consensus_queue.cc:1035] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:20.426965 30942 mvcc.cc:205] Tried to move back new op lower bound from 7299061024453746688 to 7299061024367566848. Current Snapshot: MvccSnapshot[applied={T|T < 7299061024453746688}]
I20260620 23:15:20.427089 30944 mvcc.cc:205] Tried to move back new op lower bound from 7299061024453746688 to 7299061024367566848. Current Snapshot: MvccSnapshot[applied={T|T < 7299061024453746688}]
I20260620 23:15:20.427277 30943 mvcc.cc:205] Tried to move back new op lower bound from 7299061024453746688 to 7299061024367566848. Current Snapshot: MvccSnapshot[applied={T|T < 7299061024453746688}]
I20260620 23:15:20.429378 30453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:20.429538 30453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:20.430473 30453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Sending DeleteTablet for 3 replicas of tablet 9ee87760bd2e48e29fcf81aba3c996ef
I20260620 23:15:20.430815 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 30515
I20260620 23:15:20.430816 30846 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee87760bd2e48e29fcf81aba3c996ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:20 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:54246
I20260620 23:15:20.430859 30712 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee87760bd2e48e29fcf81aba3c996ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:20 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:35692
I20260620 23:15:20.431666 30953 tablet_replica.cc:330] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1: stopping tablet replica
I20260620 23:15:20.431768 30953 raft_consensus.cc:2229] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:20.431890 30953 raft_consensus.cc:2258] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:20.432055 30952 tablet_replica.cc:330] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad: stopping tablet replica
I20260620 23:15:20.432147 30952 raft_consensus.cc:2229] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:20.432263 30952 raft_consensus.cc:2258] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:20.432876 30952 ts_tablet_manager.cc:1905] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:20.434839 30952 ts_tablet_manager.cc:1918] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:20.434911 30952 log.cc:1196] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/9ee87760bd2e48e29fcf81aba3c996ef
I20260620 23:15:20.435186 30952 ts_tablet_manager.cc:1939] T 9ee87760bd2e48e29fcf81aba3c996ef P cce78ca247f742c18dfad6ea424477ad: Deleting consensus metadata
I20260620 23:15:20.435820 30437 catalog_manager.cc:4841] TS cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389): tablet 9ee87760bd2e48e29fcf81aba3c996ef (table test-table [id=bbdf6b0d774148ba83f6298dfe533d94]) successfully deleted
W20260620 23:15:20.436736 30439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:33585 (error 108)
I20260620 23:15:20.437132 30953 ts_tablet_manager.cc:1905] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 23:15:20.438387 30923 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:15:20.437097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:33585 (local address 127.0.0.1:38834)
0620 23:15:20.437153 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.437163 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:20.437220 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:20.438344 (+ 1124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:33585: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 23:15:20.438405 30953 ts_tablet_manager.cc:1918] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:20.438476 30953 log.cc:1196] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/9ee87760bd2e48e29fcf81aba3c996ef
W20260620 23:15:20.438534 30439 catalog_manager.cc:4584] TS 11482a723bfa4625a5c84d8eb153c679 (127.16.79.1:33585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ee87760bd2e48e29fcf81aba3c996ef: Network error: Client connection negotiation failed: client connection to 127.16.79.1:33585: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:15:20.438614 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 30649
I20260620 23:15:20.438757 30953 ts_tablet_manager.cc:1939] T 9ee87760bd2e48e29fcf81aba3c996ef P 6f562397a5d54eac9ac484eaed971eb1: Deleting consensus metadata
I20260620 23:15:20.444200 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 30783
I20260620 23:15:20.444334 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.444121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34525 (local address 127.0.0.1:35708)
0620 23:15:20.444194 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.444256 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.2:34525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 23:15:20.449379 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 30421
I20260620 23:15:20.454869 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:40353
--webserver_interface=127.16.79.62
--webserver_port=41845
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:40353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:20.539551 30955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:20.539750 30955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:20.539773 30955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:20.541002 30955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:20.541040 30955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:20.541054 30955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:20.541067 30955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:20.541079 30955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:20.542493 30955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:40353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:40353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:20.542972 30955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:20.543426 30955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:15:20.551942 30955 server_base.cc:1034] running on GCE node
W20260620 23:15:20.551868 30961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:20.551869 30964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:20.551868 30962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:20.552371 30955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:20.552670 30955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:20.553857 30955 hybrid_clock.cc:648] HybridClock initialized: now 1781997320553840 us; error 38 us; skew 500 ppm
Jun 20 23:15:20 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:20.559657 30955 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:15:20.560351 30955 webserver.cc:460] Webserver started at http://127.16.79.62:41845/ using document root <none> and password file <none>
I20260620 23:15:20.560647 30955 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:20.560804 30955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:20.562651 30955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:15:20.563496 30971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.563766 30955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:20.563887 30955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "ac3dc4731ae04cada98b786e2659e5bb"
format_stamp: "Formatted at 2026-06-20 23:15:19 on dist-test-slave-pswb"
server_key: "7582e05850f649ea71b4e1eaad87d552"
server_key_iv: "64009d54fc48b05a8496b36b774b9b48"
server_key_version: "encryptionkey@0"
I20260620 23:15:20.564337 30955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:20.572302 30955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:20.574651 30955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:20.574826 30955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:20.580137 30955 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:40353
I20260620 23:15:20.580188 31023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:40353 every 8 connection(s)
I20260620 23:15:20.580648 30955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:20.582312 31024 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:15:20.582973 31024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Bootstrap starting.
I20260620 23:15:20.590170 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 30955
I20260620 23:15:20.590473 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:33585
--local_ip_for_outbound_sockets=127.16.79.1
--tserver_master_addrs=127.16.79.62:40353
--webserver_port=44513
--webserver_interface=127.16.79.1
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:20.591048 31024 log.cc:824] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:20.593677 31024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:15:20.593999 31024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Bootstrap complete.
I20260620 23:15:20.598047 31024 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } }
I20260620 23:15:20.598438 31024 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3dc4731ae04cada98b786e2659e5bb, State: Initialized, Role: FOLLOWER
I20260620 23:15:20.598613 31024 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } }
I20260620 23:15:20.598763 31024 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:20.598842 31024 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:20.598897 31024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:15:20.600656 31024 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } }
I20260620 23:15:20.600801 31024 leader_election.cc:302] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3dc4731ae04cada98b786e2659e5bb; no voters:
I20260620 23:15:20.600978 31024 leader_election.cc:288] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:15:20.601061 31028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:15:20.601274 31028 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 2 LEADER]: Becoming Leader. State: Replica: ac3dc4731ae04cada98b786e2659e5bb, State: Running, Role: LEADER
I20260620 23:15:20.601289 31024 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:20.601467 31028 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } }
I20260620 23:15:20.601842 31029 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ac3dc4731ae04cada98b786e2659e5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } } }
I20260620 23:15:20.601850 31030 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac3dc4731ae04cada98b786e2659e5bb. Latest consensus state: current_term: 2 leader_uuid: "ac3dc4731ae04cada98b786e2659e5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } } }
I20260620 23:15:20.601948 31029 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:20.601953 31030 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:20.602298 31036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:20.603181 31036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:20.603338 31036 catalog_manager.cc:1259] Loaded cluster ID: 330dc88e8ee0446f933b6e832b293b3f
I20260620 23:15:20.603781 31036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:20.604110 31036 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 23:15:20.609598 31036 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 23:15:20.612902 31036 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:20.613122 31036 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Loaded TSK: 0
I20260620 23:15:20.613363 31036 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:15:20.678555 31027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:20.678737 31027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:20.678759 31027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:20.679865 31027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:20.679904 31027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:20.679931 31027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:15:20.681386 31027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:33585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=44513
--tserver_master_addrs=127.16.79.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:20.681604 31027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:20.681813 31027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:20.687522 31052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:20.687718 31027 server_base.cc:1034] running on GCE node
W20260620 23:15:20.687515 31054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:20.687507 31051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:20.688073 31027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:20.688305 31027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:20.689472 31027 hybrid_clock.cc:648] HybridClock initialized: now 1781997320689456 us; error 31 us; skew 500 ppm
Jun 20 23:15:20 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:20.692239 31027 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:15:20.692776 31027 webserver.cc:460] Webserver started at http://127.16.79.1:44513/ using document root <none> and password file <none>
I20260620 23:15:20.693027 31027 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:20.693116 31027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:20.694689 31027 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:15:20.695456 31061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.695639 31027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:20.695791 31027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "11482a723bfa4625a5c84d8eb153c679"
format_stamp: "Formatted at 2026-06-20 23:15:19 on dist-test-slave-pswb"
server_key: "ea46306dd56975fa7b35557ff2f90d16"
server_key_iv: "0f645c1ebd12a76b94d67aeb3ff38084"
server_key_version: "encryptionkey@0"
I20260620 23:15:20.696147 31027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:20.704169 31027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:20.706879 31027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:20.707020 31027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:20.707247 31027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:20.707690 31068 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 23:15:20.709565 31027 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:15:20.709630 31027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 23:15:20.709666 31027 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:15:20.710249 31027 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:15:20.710304 31027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 23:15:20.710358 31068 tablet_bootstrap.cc:492] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Bootstrap starting.
I20260620 23:15:20.717298 31027 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:33585
I20260620 23:15:20.717634 31027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:15:20.720896 31175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:33585 every 8 connection(s)
I20260620 23:15:20.721155 31068 log.cc:824] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:20.725535 31068 tablet_bootstrap.cc:492] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:15:20.725857 31068 tablet_bootstrap.cc:492] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Bootstrap complete.
I20260620 23:15:20.726281 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 31027
I20260620 23:15:20.726567 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:34525
--local_ip_for_outbound_sockets=127.16.79.2
--tserver_master_addrs=127.16.79.62:40353
--webserver_port=40621
--webserver_interface=127.16.79.2
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:20.727368 31068 ts_tablet_manager.cc:1397] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
Jun 20 23:15:20 dist-test-slave-pswb krb5kdc[30408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:20.731231 31068 raft_consensus.cc:348] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:20.731618 31068 raft_consensus.cc:729] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11482a723bfa4625a5c84d8eb153c679, State: Initialized, Role: FOLLOWER
I20260620 23:15:20.731789 31068 consensus_queue.cc:260] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:20.732080 31068 ts_tablet_manager.cc:1428] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260620 23:15:20.733680 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.718277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:60747)
0620 23:15:20.718448 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.718488 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:20.718898 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:20.721548 (+ 2650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:20.721552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:20.721930 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:20.722464 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.722479 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.724058 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.724060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.725174 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.725178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.726079 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.726081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.726255 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.726934 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:20.726944 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:20.728987 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:20.731663 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:20.731669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:20.731671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:20.731989 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:20.732331 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:20.732332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:20.732332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:20.732464 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:20.733013 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:20.733020 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:20.733174 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 23:15:20.733298 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 23:15:20.733488 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260620 23:15:20.733798 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.718372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:60747 (local address 127.16.79.62:40353)
0620 23:15:20.721119 (+ 2747us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.721127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.721166 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:15:20.721291 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.721297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.721347 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.721486 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:20.722678 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.723857 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.725356 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.725889 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.728109 (+ 2220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.729094 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:20.729098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:20.729098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:20.729109 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:20.731558 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.732094 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.732097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.732098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.732228 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.732575 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.732575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.732576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.732925 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:20.733020 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.733372 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.733535 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2699,"thread_start_us":74,"threads_started":1}
I20260620 23:15:20.734506 31176 heartbeater.cc:347] Connected to a master server at 127.16.79.62:40353
I20260620 23:15:20.734611 31176 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:20.734803 31176 heartbeater.cc:511] Master 127.16.79.62:40353 requested a full tablet report, sending...
I20260620 23:15:20.735598 30988 ts_manager.cc:194] Registered new tserver with Master: 11482a723bfa4625a5c84d8eb153c679 (127.16.79.1:33585)
I20260620 23:15:20.736791 30988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:60747
I20260620 23:15:20.738004 31176 heartbeater.cc:503] Master 127.16.79.62:40353 was elected leader, sending a full tablet report...
Jun 20 23:15:20 dist-test-slave-pswb krb5kdc[30408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for kudu/127.16.79.1@KRBTEST.COM
I20260620 23:15:20.747543 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.736981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:33585 (local address 127.0.0.1:38838)
0620 23:15:20.737119 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.737151 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:20.737215 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:20.737554 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:20.737559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:20.737860 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:20.737919 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.737921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.739153 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.739154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.740185 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.740188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.740807 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.740808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.740902 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.741377 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:20.741382 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:20.743405 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:20.746251 (+ 2846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:20.746258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:20.746259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:20.746524 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:20.746805 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:20.746805 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:20.746806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:20.746865 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:20.747226 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:20.747231 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:20.747283 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:15:20.747341 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 23:15:20.747446 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260620 23:15:20.747944 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.737266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.16.79.1:33585)
0620 23:15:20.737396 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.737400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.737408 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:15:20.737441 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.737443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.737459 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.737996 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:20.738044 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.739069 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.740274 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.740735 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.740807 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.743582 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:20.743600 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:20.743601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:20.743611 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:20.746141 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.746673 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.746676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.746676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.746737 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.746929 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.746930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.746930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.747132 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:20.747730 (+ 598us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.747782 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.747861 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260620 23:15:20.748272 31110 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee87760bd2e48e29fcf81aba3c996ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:20 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:38838
I20260620 23:15:20.748860 31185 tablet_replica.cc:330] stopping tablet replica
I20260620 23:15:20.748971 31185 raft_consensus.cc:2229] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:20.749037 31185 raft_consensus.cc:2258] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:20.749336 31185 ts_tablet_manager.cc:1905] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:20.751493 31185 ts_tablet_manager.cc:1918] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:20.751570 31185 log.cc:1196] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal/wals/9ee87760bd2e48e29fcf81aba3c996ef
I20260620 23:15:20.751791 31185 ts_tablet_manager.cc:1939] T 9ee87760bd2e48e29fcf81aba3c996ef P 11482a723bfa4625a5c84d8eb153c679: Deleting consensus metadata
I20260620 23:15:20.752257 30974 catalog_manager.cc:4841] TS 11482a723bfa4625a5c84d8eb153c679 (127.16.79.1:33585): tablet 9ee87760bd2e48e29fcf81aba3c996ef (table test-table [id=bbdf6b0d774148ba83f6298dfe533d94]) successfully deleted
W20260620 23:15:20.814169 31181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:20.814357 31181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:20.814379 31181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:20.815429 31181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:20.815464 31181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:20.815490 31181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:15:20.816834 31181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:34525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=40621
--tserver_master_addrs=127.16.79.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:20.817001 31181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:20.817159 31181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:20.822147 31193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:20.822199 31191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:20.822216 31190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:20.822528 31181 server_base.cc:1034] running on GCE node
I20260620 23:15:20.822885 31181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:20.823146 31181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:20.824287 31181 hybrid_clock.cc:648] HybridClock initialized: now 1781997320824246 us; error 56 us; skew 500 ppm
Jun 20 23:15:20 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:20.827016 31181 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:15:20.827545 31181 webserver.cc:460] Webserver started at http://127.16.79.2:40621/ using document root <none> and password file <none>
I20260620 23:15:20.827767 31181 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:20.827847 31181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:20.829391 31181 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:15:20.830140 31200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.830479 31181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:20.830633 31181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "6f562397a5d54eac9ac484eaed971eb1"
format_stamp: "Formatted at 2026-06-20 23:15:20 on dist-test-slave-pswb"
server_key: "fba84ba67fa8ad300657e0f4ea858d70"
server_key_iv: "75ecb2da8b18884213f738407126b082"
server_key_version: "encryptionkey@0"
I20260620 23:15:20.831111 31181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:20.845988 31181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:20.848205 31181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:20.848362 31181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:20.848587 31181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:20.848948 31181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:20.848999 31181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.849035 31181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:20.849061 31181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.854746 31181 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:34525
I20260620 23:15:20.854812 31313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:34525 every 8 connection(s)
I20260620 23:15:20.855134 31181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:15:20 dist-test-slave-pswb krb5kdc[30408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:20.862859 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 31181
I20260620 23:15:20.863119 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:36389
--local_ip_for_outbound_sockets=127.16.79.3
--tserver_master_addrs=127.16.79.62:40353
--webserver_port=46557
--webserver_interface=127.16.79.3
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:20.865170 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.855839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:53665 (local address 127.16.79.62:40353)
0620 23:15:20.855892 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.855895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.856106 (+ 211us) server_negotiation.cc:407] Connection header received
0620 23:15:20.856590 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.856591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.856604 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.856652 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:20.857590 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.858090 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.859343 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.859612 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.859726 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.862002 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:20.862003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:20.862003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:20.862011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:20.863544 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.864017 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.864018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.864018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.864069 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.864353 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.864354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.864354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.864505 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:20.864611 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.864945 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.865049 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:15:20.865180 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.855834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:53665)
0620 23:15:20.855988 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.856022 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:20.856486 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:20.856779 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:20.856782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:20.857314 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:20.857496 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.857500 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.858196 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.858197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.859188 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.859191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.859738 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.859739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.859839 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.860337 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:20.860345 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:20.861891 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:20.863618 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:20.863625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:20.863627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:20.863938 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:20.864135 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:20.864135 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:20.864136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:20.864278 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:20.864611 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:20.864617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:20.864816 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 23:15:20.864913 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:15:20.865029 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":96,"threads_started":1}
I20260620 23:15:20.865758 31314 heartbeater.cc:347] Connected to a master server at 127.16.79.62:40353
I20260620 23:15:20.865836 31314 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:20.866017 31314 heartbeater.cc:511] Master 127.16.79.62:40353 requested a full tablet report, sending...
I20260620 23:15:20.866392 30988 ts_manager.cc:194] Registered new tserver with Master: 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525)
I20260620 23:15:20.866905 30988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:53665
W20260620 23:15:20.946560 31317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:20.946766 31317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:20.946789 31317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:20.947810 31317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:20.947844 31317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:20.947870 31317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:15:20.949307 31317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:36389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=46557
--tserver_master_addrs=127.16.79.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:20.949484 31317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:20.949658 31317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:20.954383 31322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:20.954386 31325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:20.954591 31317 server_base.cc:1034] running on GCE node
W20260620 23:15:20.954394 31323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:20.954834 31317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:20.955018 31317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:20.956164 31317 hybrid_clock.cc:648] HybridClock initialized: now 1781997320956151 us; error 27 us; skew 500 ppm
Jun 20 23:15:20 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:20.958524 31317 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:15:20.958932 31317 webserver.cc:460] Webserver started at http://127.16.79.3:46557/ using document root <none> and password file <none>
I20260620 23:15:20.959134 31317 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:20.959190 31317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:20.960634 31317 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:20.961596 31332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.961792 31317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:20.961899 31317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "cce78ca247f742c18dfad6ea424477ad"
format_stamp: "Formatted at 2026-06-20 23:15:20 on dist-test-slave-pswb"
server_key: "699176874c5f29785f596fa795661173"
server_key_iv: "cff7cb8c1a59a7f9fdfa6f0e0d10518a"
server_key_version: "encryptionkey@0"
I20260620 23:15:20.962249 31317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:20.978880 31317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:20.981051 31317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:20.981202 31317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:20.981412 31317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:20.981707 31317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:20.981760 31317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.981801 31317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:20.981828 31317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.987550 31317 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:36389
I20260620 23:15:20.987671 31445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:36389 every 8 connection(s)
I20260620 23:15:20.987948 31317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:15:20.988771 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 31317
Jun 20 23:15:20 dist-test-slave-pswb krb5kdc[30408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:20.995710 31449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.989215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.16.79.62:40353)
0620 23:15:20.989327 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.989329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.989480 (+ 151us) server_negotiation.cc:407] Connection header received
0620 23:15:20.989569 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.989570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.989581 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.989647 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:20.990278 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.990781 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.991503 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.991722 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.991804 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.993068 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:20.993070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:20.993070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:20.993077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:20.994504 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.994866 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.994867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.994868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.994944 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.995151 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.995152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.995152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.995270 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:20.995389 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.995502 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.995588 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20260620 23:15:20.997684 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.988781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:35599 (local address 127.16.79.62:40353)
0620 23:15:20.988856 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:20.988859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:20.989117 (+ 258us) server_negotiation.cc:407] Connection header received
0620 23:15:20.989844 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:20.989845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:20.989855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:20.989914 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:20.990785 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.991255 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.992239 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.992443 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.992523 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.994732 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:20.994734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:20.994734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:20.994741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:20.996332 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.996789 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.996790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.996790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.996843 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:20.997103 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:20.997104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:20.997104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:20.997206 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:20.997273 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:20.997532 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 23:15:20.997606 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:15:20.997864 31448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:20.988613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:35599)
0620 23:15:20.988869 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:20.988966 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:20.989741 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:20.989977 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:20.989980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:20.990509 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:20.990683 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.990686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.991346 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.991347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.992123 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:20.992125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:20.992534 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:20.992535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:20.992653 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:20.993100 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:20.993110 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:20.994628 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:20.996413 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:20.996419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:20.996421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:20.996729 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:20.996922 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:20.996923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:20.996924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:20.997034 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:20.997267 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:20.997273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:20.997401 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 23:15:20.997498 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:15:20.997642 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":121,"threads_started":1}
I20260620 23:15:20.998302 31446 heartbeater.cc:347] Connected to a master server at 127.16.79.62:40353
I20260620 23:15:20.998374 31446 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:20.998500 31446 heartbeater.cc:511] Master 127.16.79.62:40353 requested a full tablet report, sending...
I20260620 23:15:20.998755 30988 ts_manager.cc:194] Registered new tserver with Master: cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389)
I20260620 23:15:20.999187 30988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:35599
I20260620 23:15:21.007416 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:15:21.010864 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.007774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57524 (local address 127.16.79.62:40353)
0620 23:15:21.007840 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:21.007842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:21.007986 (+ 144us) server_negotiation.cc:407] Connection header received
0620 23:15:21.008038 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:21.008039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:21.008070 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:21.008116 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:21.008230 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.008721 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.009869 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.010085 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.010169 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:21.010406 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:21.010540 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:15:21.010628 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:21.010696 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 23:15:21.010717 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:15:21.011264 30988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:15:21.014019 30988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:15:21.016047 31110 tablet_service.cc:1467] Processing CreateTablet for tablet 99a6d0f8afd848d8891f15fb06b4ddfd (DEFAULT_TABLE table=test-table [id=e2d0539b2044482a9002a057d60a61b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:21.016252 31110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a6d0f8afd848d8891f15fb06b4ddfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:21.017855 31068 tablet_bootstrap.cc:492] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: Bootstrap starting.
I20260620 23:15:21.018180 31068 tablet_bootstrap.cc:654] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:21.019008 31068 tablet_bootstrap.cc:492] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: No bootstrap required, opened a new log
I20260620 23:15:21.019061 31068 ts_tablet_manager.cc:1397] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:21.019212 31068 raft_consensus.cc:348] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:21.019302 31068 raft_consensus.cc:374] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:21.019351 31068 raft_consensus.cc:729] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11482a723bfa4625a5c84d8eb153c679, State: Initialized, Role: FOLLOWER
I20260620 23:15:21.019395 31068 consensus_queue.cc:260] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:21.019484 31068 ts_tablet_manager.cc:1428] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:15:21.020303 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.016003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35712 (local address 127.16.79.2:34525)
0620 23:15:21.016204 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:21.016209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:21.016220 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:15:21.016259 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:21.016262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:21.016277 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:21.016395 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:21.016684 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.017820 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.019263 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.019988 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.020062 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:21.020123 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:21.020214 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 23:15:21.020244 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
I20260620 23:15:21.020313 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.015786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34525 (local address 127.0.0.1:35712)
0620 23:15:21.015854 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.015865 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:21.015899 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:21.016411 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:21.016412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:21.016414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:21.016462 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:21.016464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.017933 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.017934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:21.019179 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:21.019182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.020063 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.020063 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:21.020125 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:21.020144 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 23:15:21.020205 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:15:21.020246 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":12}
I20260620 23:15:21.021136 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.015842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:36389 (local address 127.0.0.1:54248)
0620 23:15:21.016010 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.016021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:21.016063 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:21.016560 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:21.016561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:21.016563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:21.016601 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:21.016603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.017637 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.017638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:21.019995 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:21.019998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.020915 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.020916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:21.020979 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:21.020982 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:15:21.021032 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 23:15:21.021069 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":30,"thread_start_us":111,"threads_started":1}
I20260620 23:15:21.021148 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.016247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54248 (local address 127.16.79.3:36389)
0620 23:15:21.016421 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:21.016425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:21.016434 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:15:21.016468 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:21.016470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:21.016485 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:21.016561 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:21.016678 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.017535 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.020085 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.020827 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.020902 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:21.020951 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:21.021041 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 23:15:21.021074 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260620 23:15:21.021028 31248 tablet_service.cc:1467] Processing CreateTablet for tablet 99a6d0f8afd848d8891f15fb06b4ddfd (DEFAULT_TABLE table=test-table [id=e2d0539b2044482a9002a057d60a61b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:21.021381 31248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a6d0f8afd848d8891f15fb06b4ddfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:21.024008 31456 tablet_bootstrap.cc:492] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1: Bootstrap starting.
I20260620 23:15:21.024554 31380 tablet_service.cc:1467] Processing CreateTablet for tablet 99a6d0f8afd848d8891f15fb06b4ddfd (DEFAULT_TABLE table=test-table [id=e2d0539b2044482a9002a057d60a61b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:21.024693 31456 tablet_bootstrap.cc:654] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:21.024850 31380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a6d0f8afd848d8891f15fb06b4ddfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:21.025243 31456 log.cc:824] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:21.026291 31456 tablet_bootstrap.cc:492] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1: No bootstrap required, opened a new log
I20260620 23:15:21.026451 31456 ts_tablet_manager.cc:1397] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:15:21.027532 31458 tablet_bootstrap.cc:492] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad: Bootstrap starting.
I20260620 23:15:21.028421 31458 tablet_bootstrap.cc:654] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:21.028405 31456 raft_consensus.cc:348] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:21.028628 31456 raft_consensus.cc:374] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:21.028680 31456 raft_consensus.cc:729] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f562397a5d54eac9ac484eaed971eb1, State: Initialized, Role: FOLLOWER
I20260620 23:15:21.028852 31456 consensus_queue.cc:260] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:21.029130 31458 log.cc:824] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:21.029263 31314 heartbeater.cc:503] Master 127.16.79.62:40353 was elected leader, sending a full tablet report...
I20260620 23:15:21.029232 31456 ts_tablet_manager.cc:1428] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:15:21.030018 31458 tablet_bootstrap.cc:492] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad: No bootstrap required, opened a new log
I20260620 23:15:21.030102 31458 ts_tablet_manager.cc:1397] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:15:21.031896 31458 raft_consensus.cc:348] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:21.032071 31458 raft_consensus.cc:374] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:21.032109 31458 raft_consensus.cc:729] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce78ca247f742c18dfad6ea424477ad, State: Initialized, Role: FOLLOWER
I20260620 23:15:21.032290 31458 consensus_queue.cc:260] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:21.032668 31458 ts_tablet_manager.cc:1428] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:15:21.032729 31446 heartbeater.cc:503] Master 127.16.79.62:40353 was elected leader, sending a full tablet report...
W20260620 23:15:21.105685 31315 tablet.cc:2369] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:21.123869 31461 raft_consensus.cc:482] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:21.124039 31461 raft_consensus.cc:504] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:21.124437 31461 leader_election.cc:288] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525), 11482a723bfa4625a5c84d8eb153c679 (127.16.79.1:33585)
I20260620 23:15:21.128906 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.124681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:46321 (local address 127.16.79.2:34525)
0620 23:15:21.124750 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:21.124752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:21.124788 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:15:21.124819 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:21.124820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:21.124832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:21.124907 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:21.125104 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.125845 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.127664 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.128339 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.128474 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:21.128493 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:21.128773 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 23:15:21.128839 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:15:21.128917 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.124740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:57023 (local address 127.16.79.1:33585)
0620 23:15:21.124808 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:21.124810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:21.124891 (+ 81us) server_negotiation.cc:407] Connection header received
0620 23:15:21.124959 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:21.124960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:21.124985 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:21.125032 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:21.125191 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.125905 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.127665 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.128456 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.128530 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:21.128609 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:21.128773 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 23:15:21.128857 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:15:21.128902 31448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.124597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34525 (local address 127.16.79.3:46321)
0620 23:15:21.124655 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.124671 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:21.124739 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:21.124929 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:21.124930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:21.124932 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:21.125007 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:21.125009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.126047 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.126049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:21.127521 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:21.127523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.128486 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.128487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:21.128626 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:21.128634 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:15:21.128743 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:15:21.128803 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":212}
I20260620 23:15:21.128913 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.124620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:33585 (local address 127.16.79.3:57023)
0620 23:15:21.124814 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.124823 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:21.124875 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:21.125057 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:21.125058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:21.125060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:21.125119 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:21.125120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.126046 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.126049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:21.127514 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:21.127519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.128556 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.128556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:21.128626 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:21.128635 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:15:21.128743 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:15:21.128802 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":62,"thread_start_us":121,"threads_started":1}
I20260620 23:15:21.129295 31268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a6d0f8afd848d8891f15fb06b4ddfd" candidate_uuid: "cce78ca247f742c18dfad6ea424477ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f562397a5d54eac9ac484eaed971eb1" is_pre_election: true
I20260620 23:15:21.129452 31268 raft_consensus.cc:2454] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cce78ca247f742c18dfad6ea424477ad in term 0.
I20260620 23:15:21.129523 31130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a6d0f8afd848d8891f15fb06b4ddfd" candidate_uuid: "cce78ca247f742c18dfad6ea424477ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11482a723bfa4625a5c84d8eb153c679" is_pre_election: true
I20260620 23:15:21.129689 31130 raft_consensus.cc:2454] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cce78ca247f742c18dfad6ea424477ad in term 0.
I20260620 23:15:21.129704 31334 leader_election.cc:302] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f562397a5d54eac9ac484eaed971eb1, cce78ca247f742c18dfad6ea424477ad; no voters:
I20260620 23:15:21.129832 31461 raft_consensus.cc:2781] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:15:21.129900 31461 raft_consensus.cc:482] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:15:21.129933 31461 raft_consensus.cc:3037] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:21.130959 31461 raft_consensus.cc:504] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:21.131210 31461 leader_election.cc:288] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [CANDIDATE]: Term 1 election: Requested vote from peers 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525), 11482a723bfa4625a5c84d8eb153c679 (127.16.79.1:33585)
I20260620 23:15:21.131446 31130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a6d0f8afd848d8891f15fb06b4ddfd" candidate_uuid: "cce78ca247f742c18dfad6ea424477ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11482a723bfa4625a5c84d8eb153c679"
I20260620 23:15:21.131393 31268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a6d0f8afd848d8891f15fb06b4ddfd" candidate_uuid: "cce78ca247f742c18dfad6ea424477ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f562397a5d54eac9ac484eaed971eb1"
I20260620 23:15:21.131556 31130 raft_consensus.cc:3037] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:21.131577 31268 raft_consensus.cc:3037] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:21.132925 31268 raft_consensus.cc:2454] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cce78ca247f742c18dfad6ea424477ad in term 1.
I20260620 23:15:21.132937 31130 raft_consensus.cc:2454] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cce78ca247f742c18dfad6ea424477ad in term 1.
I20260620 23:15:21.133095 31334 leader_election.cc:302] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f562397a5d54eac9ac484eaed971eb1, cce78ca247f742c18dfad6ea424477ad; no voters:
I20260620 23:15:21.133196 31461 raft_consensus.cc:2781] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:21.133338 31461 raft_consensus.cc:686] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 1 LEADER]: Becoming Leader. State: Replica: cce78ca247f742c18dfad6ea424477ad, State: Running, Role: LEADER
I20260620 23:15:21.133441 31461 consensus_queue.cc:237] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:21.134259 30988 catalog_manager.cc:5495] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad reported cstate change: term changed from 0 to 1, leader changed from <none> to cce78ca247f742c18dfad6ea424477ad (127.16.79.3). New cstate: current_term: 1 leader_uuid: "cce78ca247f742c18dfad6ea424477ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } health_report { overall_health: UNKNOWN } } }
I20260620 23:15:21.166363 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.163969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.16.79.3:36389)
0620 23:15:21.164041 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:21.164043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:21.164051 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:15:21.164098 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:21.164098 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:21.164128 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:21.164205 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:21.164310 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.164880 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.165615 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:21.165799 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:21.165878 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:21.166044 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:21.166151 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:15:21.166241 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:21.166260 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 23:15:21.166278 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:15:21.168210 31268 raft_consensus.cc:1261] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1 [term 1 FOLLOWER]: Refusing update from remote peer cce78ca247f742c18dfad6ea424477ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:21.168263 31130 raft_consensus.cc:1261] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Refusing update from remote peer cce78ca247f742c18dfad6ea424477ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:21.168562 31463 consensus_queue.cc:1035] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:21.168778 31461 consensus_queue.cc:1035] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:21.171254 31466 mvcc.cc:205] Tried to move back new op lower bound from 7299061027502600192 to 7299061027363246080. Current Snapshot: MvccSnapshot[applied={T|T < 7299061027502600192}]
I20260620 23:15:21.174633 30988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:21.174739 30988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:21.175459 30988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Sending DeleteTablet for 3 replicas of tablet 99a6d0f8afd848d8891f15fb06b4ddfd
I20260620 23:15:21.175776 31380 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a6d0f8afd848d8891f15fb06b4ddfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:21 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:54248
I20260620 23:15:21.175818 31248 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a6d0f8afd848d8891f15fb06b4ddfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:21 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:35712
I20260620 23:15:21.176061 31476 tablet_replica.cc:330] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1: stopping tablet replica
I20260620 23:15:21.176028 31110 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a6d0f8afd848d8891f15fb06b4ddfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:21 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:38838
I20260620 23:15:21.176177 31476 raft_consensus.cc:2229] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:21.176200 31185 tablet_replica.cc:330] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: stopping tablet replica
I20260620 23:15:21.176270 31185 raft_consensus.cc:2229] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:21.176294 31476 raft_consensus.cc:2258] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:21.176331 31185 raft_consensus.cc:2258] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:21.176618 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 31027
I20260620 23:15:21.176857 31185 ts_tablet_manager.cc:1905] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:21.177222 31476 ts_tablet_manager.cc:1905] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:21.177557 31475 tablet_replica.cc:330] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad: stopping tablet replica
I20260620 23:15:21.177661 31475 raft_consensus.cc:2229] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:15:21.177784 31475 raft_consensus.cc:2258] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:21.179006 31476 ts_tablet_manager.cc:1918] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:21.179105 31476 log.cc:1196] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/99a6d0f8afd848d8891f15fb06b4ddfd
I20260620 23:15:21.179386 31476 ts_tablet_manager.cc:1939] T 99a6d0f8afd848d8891f15fb06b4ddfd P 6f562397a5d54eac9ac484eaed971eb1: Deleting consensus metadata
I20260620 23:15:21.179801 30973 catalog_manager.cc:4841] TS 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525): tablet 99a6d0f8afd848d8891f15fb06b4ddfd (table test-table [id=e2d0539b2044482a9002a057d60a61b3]) successfully deleted
I20260620 23:15:21.181321 31475 ts_tablet_manager.cc:1905] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 23:15:21.183470 30974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:33585 (error 108)
I20260620 23:15:21.183938 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 31181
I20260620 23:15:21.183954 31475 ts_tablet_manager.cc:1918] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:21.184023 31475 log.cc:1196] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/99a6d0f8afd848d8891f15fb06b4ddfd
I20260620 23:15:21.184541 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.184058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:33585 (local address 127.0.0.1:38842)
0620 23:15:21.184469 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.184504 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:33585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
I20260620 23:15:21.184882 31475 ts_tablet_manager.cc:1939] T 99a6d0f8afd848d8891f15fb06b4ddfd P cce78ca247f742c18dfad6ea424477ad: Deleting consensus metadata
I20260620 23:15:21.185432 30972 catalog_manager.cc:4841] TS cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389): tablet 99a6d0f8afd848d8891f15fb06b4ddfd (table test-table [id=e2d0539b2044482a9002a057d60a61b3]) successfully deleted
W20260620 23:15:21.185491 30974 catalog_manager.cc:4584] TS 11482a723bfa4625a5c84d8eb153c679 (127.16.79.1:33585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99a6d0f8afd848d8891f15fb06b4ddfd: Network error: Client connection negotiation failed: client connection to 127.16.79.1:33585: connect: Connection refused (error 111)
I20260620 23:15:21.189611 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 31317
I20260620 23:15:21.195266 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 30955
I20260620 23:15:21.201625 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:40353
--webserver_interface=127.16.79.62
--webserver_port=41845
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:40353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:21.281935 31478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:21.282384 31478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:21.282490 31478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:21.283645 31478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:21.283712 31478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:21.283741 31478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:21.283773 31478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:21.283799 31478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:21.285319 31478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:40353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:40353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:21.285568 31478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:21.285776 31478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:21.290421 31486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:21.290395 31483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:21.290413 31484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:21.290751 31478 server_base.cc:1034] running on GCE node
I20260620 23:15:21.291049 31478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:21.291273 31478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:21.292421 31478 hybrid_clock.cc:648] HybridClock initialized: now 1781997321292411 us; error 26 us; skew 500 ppm
Jun 20 23:15:21 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:21.294972 31478 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:15:21.295423 31478 webserver.cc:460] Webserver started at http://127.16.79.62:41845/ using document root <none> and password file <none>
I20260620 23:15:21.295640 31478 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:21.295686 31478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:21.297075 31478 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:21.297730 31493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.297904 31478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:15:21.297989 31478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "ac3dc4731ae04cada98b786e2659e5bb"
format_stamp: "Formatted at 2026-06-20 23:15:19 on dist-test-slave-pswb"
server_key: "7582e05850f649ea71b4e1eaad87d552"
server_key_iv: "64009d54fc48b05a8496b36b774b9b48"
server_key_version: "encryptionkey@0"
I20260620 23:15:21.298305 31478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:21.309978 31478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:21.311735 31478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:21.311874 31478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:21.316128 31478 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:40353
I20260620 23:15:21.316164 31545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:40353 every 8 connection(s)
I20260620 23:15:21.316519 31478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:21.316989 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 31478
I20260620 23:15:21.317356 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:33585
--local_ip_for_outbound_sockets=127.16.79.1
--tserver_master_addrs=127.16.79.62:40353
--webserver_port=44513
--webserver_interface=127.16.79.1
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:21.318850 31546 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:15:21.319482 31546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Bootstrap starting.
I20260620 23:15:21.327490 31546 log.cc:824] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:21.329708 31546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:15:21.329975 31546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb: Bootstrap complete.
I20260620 23:15:21.332154 31546 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } }
I20260620 23:15:21.332384 31546 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3dc4731ae04cada98b786e2659e5bb, State: Initialized, Role: FOLLOWER
I20260620 23:15:21.332542 31546 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } }
I20260620 23:15:21.332598 31546 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:21.332659 31546 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:21.332724 31546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:15:21.333662 31546 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } }
I20260620 23:15:21.333770 31546 leader_election.cc:302] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3dc4731ae04cada98b786e2659e5bb; no voters:
I20260620 23:15:21.333949 31546 leader_election.cc:288] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:15:21.333981 31550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:15:21.334084 31550 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [term 3 LEADER]: Becoming Leader. State: Replica: ac3dc4731ae04cada98b786e2659e5bb, State: Running, Role: LEADER
I20260620 23:15:21.334174 31550 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } }
I20260620 23:15:21.334211 31546 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:21.334410 31550 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac3dc4731ae04cada98b786e2659e5bb. Latest consensus state: current_term: 3 leader_uuid: "ac3dc4731ae04cada98b786e2659e5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } } }
I20260620 23:15:21.334486 31550 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:21.334414 31551 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ac3dc4731ae04cada98b786e2659e5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3dc4731ae04cada98b786e2659e5bb" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40353 } } }
I20260620 23:15:21.334756 31551 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3dc4731ae04cada98b786e2659e5bb [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:21.334676 31554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:21.335561 31554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:21.335665 31554 catalog_manager.cc:1259] Loaded cluster ID: 330dc88e8ee0446f933b6e832b293b3f
I20260620 23:15:21.335695 31554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 23:15:21.335912 31554 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa16bd6d9ad google::LogMessage::Fail() at ??:0
@ 0x7fa16bd7088a google::LogMessage::SendToLog() at ??:0
@ 0x7fa16bd6d4de google::LogMessage::Flush() at ??:0
@ 0x7fa16bd6ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa173c8e0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa173c978b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa16dc3b0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa16dc30954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa16ea946db start_thread at ??:0
@ 0x7fa16a86b61f clone at ??:0
W20260620 23:15:21.414345 31548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:21.414747 31548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:21.414809 31548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:21.416601 31548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:21.416672 31548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:21.416761 31548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:15:21.419556 31548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:33585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=44513
--tserver_master_addrs=127.16.79.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:21.419859 31548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:21.420105 31548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:21.425046 31583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:21.425055 31582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:21.425249 31548 server_base.cc:1034] running on GCE node
W20260620 23:15:21.425377 31585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:21.425587 31548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:21.425844 31548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:21.427013 31548 hybrid_clock.cc:648] HybridClock initialized: now 1781997321426999 us; error 29 us; skew 500 ppm
Jun 20 23:15:21 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:21.429859 31548 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:15:21.430581 31548 webserver.cc:460] Webserver started at http://127.16.79.1:44513/ using document root <none> and password file <none>
I20260620 23:15:21.430907 31548 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:21.430962 31548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:21.432577 31548 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:15:21.433413 31592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.433588 31548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:21.433701 31548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "11482a723bfa4625a5c84d8eb153c679"
format_stamp: "Formatted at 2026-06-20 23:15:19 on dist-test-slave-pswb"
server_key: "ea46306dd56975fa7b35557ff2f90d16"
server_key_iv: "0f645c1ebd12a76b94d67aeb3ff38084"
server_key_version: "encryptionkey@0"
I20260620 23:15:21.434052 31548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:21.441423 31548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:21.443467 31548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:21.443609 31548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:21.443814 31548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:21.444209 31599 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 23:15:21.445838 31548 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:15:21.445896 31548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 23:15:21.445937 31548 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:15:21.446511 31548 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:15:21.446561 31548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 23:15:21.446695 31599 tablet_bootstrap.cc:492] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: Bootstrap starting.
I20260620 23:15:21.448563 31599 log.cc:824] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:21.452761 31599 tablet_bootstrap.cc:492] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:15:21.453059 31599 tablet_bootstrap.cc:492] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: Bootstrap complete.
I20260620 23:15:21.453233 31599 ts_tablet_manager.cc:1397] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 23:15:21.454296 31548 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:33585
I20260620 23:15:21.454417 31707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:33585 every 8 connection(s)
I20260620 23:15:21.454663 31548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:15:21.455307 31599 raft_consensus.cc:348] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:21.455808 31599 raft_consensus.cc:729] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11482a723bfa4625a5c84d8eb153c679, State: Initialized, Role: FOLLOWER
I20260620 23:15:21.456037 31599 consensus_queue.cc:260] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:21.456480 31599 ts_tablet_manager.cc:1428] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 23:15:21.458760 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 31548
I20260620 23:15:21.459014 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:34525
--local_ip_for_outbound_sockets=127.16.79.2
--tserver_master_addrs=127.16.79.62:40353
--webserver_port=40621
--webserver_interface=127.16.79.2
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:21.543676 31712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:21.543937 31712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:21.543979 31712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:21.545449 31712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:21.545516 31712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:21.545567 31712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:15:21.547300 31712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:34525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=40621
--tserver_master_addrs=127.16.79.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:21.547616 31712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:21.548028 31712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:21.553447 31717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:21.553427 31720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:21.553481 31718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:21.553771 31712 server_base.cc:1034] running on GCE node
I20260620 23:15:21.553987 31712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:21.554267 31712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:21.555436 31712 hybrid_clock.cc:648] HybridClock initialized: now 1781997321555422 us; error 26 us; skew 500 ppm
Jun 20 23:15:21 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:21.558463 31712 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:15:21.558974 31712 webserver.cc:460] Webserver started at http://127.16.79.2:40621/ using document root <none> and password file <none>
I20260620 23:15:21.559232 31712 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:21.559296 31712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:21.561017 31712 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:15:21.561724 31727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.561910 31712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:21.561990 31712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "6f562397a5d54eac9ac484eaed971eb1"
format_stamp: "Formatted at 2026-06-20 23:15:20 on dist-test-slave-pswb"
server_key: "fba84ba67fa8ad300657e0f4ea858d70"
server_key_iv: "75ecb2da8b18884213f738407126b082"
server_key_version: "encryptionkey@0"
I20260620 23:15:21.562347 31712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260620 23:15:21.577359 31710 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:15:21.455265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:57345)
0620 23:15:21.455414 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.455458 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:21.455597 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:21.577160 (+121563us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
W20260620 23:15:21.577589 31708 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:40353 (0 consecutive failures): Network error: Failed to ping master at 127.16.79.62:40353: Client connection negotiation failed: client connection to 127.16.79.62:40353: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:15:21.577872 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.577772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:58581)
0620 23:15:21.577820 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.577854 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 23:15:21.578140 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.578052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:57365)
0620 23:15:21.578090 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.578111 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 23:15:21.578239 31708 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:15:21.595800 31712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:21.598418 31712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:21.598529 31712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:21.598721 31712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:21.599017 31712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:21.599050 31712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.599102 31712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:21.599130 31712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.604763 31712 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:34525
I20260620 23:15:21.605036 31840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:34525 every 8 connection(s)
I20260620 23:15:21.605568 31712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:15:21.606830 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.606337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:34539)
0620 23:15:21.606547 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.606657 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
W20260620 23:15:21.607479 31841 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:40353 (0 consecutive failures): Network error: Failed to ping master at 127.16.79.62:40353: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
I20260620 23:15:21.607944 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.607794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:59675)
0620 23:15:21.607882 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.607916 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:15:21.608458 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.608340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:35185)
0620 23:15:21.608391 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.608414 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 23:15:21.608595 31841 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:15:21.614883 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 31712
I20260620 23:15:21.615125 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:36389
--local_ip_for_outbound_sockets=127.16.79.3
--tserver_master_addrs=127.16.79.62:40353
--webserver_port=46557
--webserver_interface=127.16.79.3
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:21.700528 31844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:21.700748 31844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:21.700773 31844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:21.701893 31844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:21.701943 31844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:21.701972 31844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:15:21.703377 31844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:36389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=46557
--tserver_master_addrs=127.16.79.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:21.703617 31844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:21.703856 31844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:21.708755 31852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:21.708808 31849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:21.708827 31850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:21.709009 31844 server_base.cc:1034] running on GCE node
I20260620 23:15:21.709232 31844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:21.709470 31844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:21.710630 31844 hybrid_clock.cc:648] HybridClock initialized: now 1781997321710610 us; error 31 us; skew 500 ppm
Jun 20 23:15:21 dist-test-slave-pswb krb5kdc[30408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:21.713722 31844 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:15:21.714188 31844 webserver.cc:460] Webserver started at http://127.16.79.3:46557/ using document root <none> and password file <none>
I20260620 23:15:21.714414 31844 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:21.714490 31844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:21.715804 31844 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:21.716454 31859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.716635 31844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:21.716755 31844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "cce78ca247f742c18dfad6ea424477ad"
format_stamp: "Formatted at 2026-06-20 23:15:20 on dist-test-slave-pswb"
server_key: "699176874c5f29785f596fa795661173"
server_key_iv: "cff7cb8c1a59a7f9fdfa6f0e0d10518a"
server_key_version: "encryptionkey@0"
I20260620 23:15:21.717095 31844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:21.730568 31844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:21.732548 31844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:21.732720 31844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:21.732935 31844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:21.733228 31844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:21.733281 31844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.733315 31844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:21.733342 31844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.740020 31844 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:36389
I20260620 23:15:21.740091 31972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:36389 every 8 connection(s)
I20260620 23:15:21.740409 31844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:15:21.741595 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.741030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:45427)
0620 23:15:21.741355 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.741436 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
W20260620 23:15:21.742043 31973 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:40353 (0 consecutive failures): Network error: Failed to ping master at 127.16.79.62:40353: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
I20260620 23:15:21.742357 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.742253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:48929)
0620 23:15:21.742312 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.742339 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:15:21.742597 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.742549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:34033)
0620 23:15:21.742569 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.742586 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260620 23:15:21.742693 31973 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:15:21.750411 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 31844
I20260620 23:15:22.578775 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:22.578560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:49511)
0620 23:15:22.578684 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:22.578717 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260620 23:15:22.609098 31978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:22.608909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:45117)
0620 23:15:22.609011 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:22.609056 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260620 23:15:22.743497 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:22.743192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:45383)
0620 23:15:22.743350 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:22.743399 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260620 23:15:22.763095 31980 raft_consensus.cc:482] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:22.763204 31980 raft_consensus.cc:504] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:22.763540 31980 leader_election.cc:288] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525), cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389)
Jun 20 23:15:22 dist-test-slave-pswb krb5kdc[30408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.3@KRBTEST.COM
Jun 20 23:15:22 dist-test-slave-pswb krb5kdc[30408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.2@KRBTEST.COM
I20260620 23:15:22.775135 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:22.763867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:37037 (local address 127.16.79.3:36389)
0620 23:15:22.764010 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:22.764043 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:22.764071 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:15:22.764771 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:22.764774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:22.764823 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:22.764969 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:22.765671 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:22.766593 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:22.767536 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:22.767928 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:22.768058 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:22.770411 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:22.770415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:22.770415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:22.770424 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:22.772374 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:22.773288 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:22.773291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:22.773291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:22.773384 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:22.773781 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:22.773783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:22.773784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:22.774352 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:22.774647 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:22.774873 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 23:15:22.775015 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260620 23:15:22.775144 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:22.763867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:50461 (local address 127.16.79.2:34525)
0620 23:15:22.764031 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:22.764054 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:22.764098 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:15:22.764671 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:22.764673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:22.764761 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:22.764912 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:22.765911 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:22.766802 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:22.767542 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:22.767930 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:22.768056 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:22.770802 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:22.770805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:22.770805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:22.770814 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:22.772610 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:22.773238 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:22.773240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:22.773241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:22.773350 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:22.773781 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:22.773783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:22.773784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:22.774357 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:22.774474 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:22.774930 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 23:15:22.775048 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260620 23:15:22.775607 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:22.763726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:34525 (local address 127.16.79.1:50461)
0620 23:15:22.763794 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:22.763806 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:22.763865 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:22.764946 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:22.764949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:22.765345 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:22.765807 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:22.765810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:22.766899 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:22.766900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:22.767447 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:22.767449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:22.768031 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:22.768032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:22.768154 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:22.768794 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:22.768802 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:22.770693 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:22.772772 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:22.772772 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:22.772773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:22.773136 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:22.773479 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:22.773480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:22.773481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:22.773647 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:22.774467 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:22.774475 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:22.774604 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 23:15:22.774737 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 23:15:22.775029 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":48}
I20260620 23:15:22.775609 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:22.763726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:36389 (local address 127.16.79.1:37037)
0620 23:15:22.763885 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:22.763895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:22.763936 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:22.764957 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:22.764959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:22.765347 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:22.765557 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:22.765563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:22.766695 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:22.766696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:22.767448 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:22.767450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:22.768036 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:22.768037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:22.768155 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:22.768852 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:22.768857 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:22.770311 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:22.772520 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:22.772536 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:22.772539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:22.773110 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:22.773513 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:22.773513 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:22.773514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:22.773647 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:22.774641 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:22.774641 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:22.774683 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:15:22.774754 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 23:15:22.775092 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":233,"thread_start_us":53,"threads_started":1}
W20260620 23:15:22.776613 31593 leader_election.cc:341] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389): Not found: Tablet not found: 99a6d0f8afd848d8891f15fb06b4ddfd
W20260620 23:15:22.776772 31594 leader_election.cc:341] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525): Not found: Tablet not found: 99a6d0f8afd848d8891f15fb06b4ddfd
I20260620 23:15:22.776978 31593 leader_election.cc:302] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11482a723bfa4625a5c84d8eb153c679; no voters: 6f562397a5d54eac9ac484eaed971eb1, cce78ca247f742c18dfad6ea424477ad
I20260620 23:15:22.777163 31980 raft_consensus.cc:2726] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:15:23.579685 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:23.579413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:42195)
0620 23:15:23.579584 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:23.579629 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20260620 23:15:23.609946 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:23.609665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:43963)
0620 23:15:23.609846 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:23.609892 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":114,"threads_started":1}
I20260620 23:15:23.744161 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:23.743954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:58381)
0620 23:15:23.744086 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:23.744125 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260620 23:15:24.580489 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:24.580269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:47189)
0620 23:15:24.580401 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:24.580440 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260620 23:15:24.610726 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:24.610480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:46135)
0620 23:15:24.610640 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:24.610680 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260620 23:15:24.639392 31989 raft_consensus.cc:482] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:24.639482 31989 raft_consensus.cc:504] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:24.639711 31989 leader_election.cc:288] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525), cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389)
W20260620 23:15:24.640107 31594 leader_election.cc:341] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525): Not found: Tablet not found: 99a6d0f8afd848d8891f15fb06b4ddfd
W20260620 23:15:24.640182 31593 leader_election.cc:341] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389): Not found: Tablet not found: 99a6d0f8afd848d8891f15fb06b4ddfd
I20260620 23:15:24.640254 31593 leader_election.cc:302] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11482a723bfa4625a5c84d8eb153c679; no voters: 6f562397a5d54eac9ac484eaed971eb1, cce78ca247f742c18dfad6ea424477ad
I20260620 23:15:24.640336 31989 raft_consensus.cc:2726] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:15:24.745070 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:24.744673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:38305)
0620 23:15:24.744999 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:24.745035 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":230,"threads_started":1}
I20260620 23:15:25.581089 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:25.580882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:35273)
0620 23:15:25.581010 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:25.581043 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260620 23:15:25.611392 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:25.611190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:52629)
0620 23:15:25.611323 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:25.611357 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
I20260620 23:15:25.745669 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:25.745467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:33389)
0620 23:15:25.745600 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:25.745633 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260620 23:15:26.320094 31994 raft_consensus.cc:482] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:26.320228 31994 raft_consensus.cc:504] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:26.320478 31994 leader_election.cc:288] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525), cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389)
W20260620 23:15:26.320996 31593 leader_election.cc:341] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389): Not found: Tablet not found: 99a6d0f8afd848d8891f15fb06b4ddfd
W20260620 23:15:26.321163 31594 leader_election.cc:341] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525): Not found: Tablet not found: 99a6d0f8afd848d8891f15fb06b4ddfd
I20260620 23:15:26.321233 31594 leader_election.cc:302] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11482a723bfa4625a5c84d8eb153c679; no voters: 6f562397a5d54eac9ac484eaed971eb1, cce78ca247f742c18dfad6ea424477ad
I20260620 23:15:26.321382 31994 raft_consensus.cc:2726] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:15:26.581918 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:26.581599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:53009)
0620 23:15:26.581795 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:26.581851 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260620 23:15:26.612107 31996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:26.611880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:59563)
0620 23:15:26.612030 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:26.612066 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260620 23:15:26.746382 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:26.746178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:55293)
0620 23:15:26.746309 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:26.746344 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260620 23:15:27.582743 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:27.582517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:36803)
0620 23:15:27.582662 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:27.582695 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260620 23:15:27.612920 31999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:27.612521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:47119)
0620 23:15:27.612803 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:27.612859 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":76,"threads_started":1}
I20260620 23:15:27.747035 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:27.746818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:46527)
0620 23:15:27.746964 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:27.746998 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":59,"threads_started":1}
I20260620 23:15:28.490700 32001 raft_consensus.cc:482] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:28.490789 32001 raft_consensus.cc:504] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:28.490959 32001 leader_election.cc:288] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525), cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389)
W20260620 23:15:28.491345 31593 leader_election.cc:341] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389): Not found: Tablet not found: 99a6d0f8afd848d8891f15fb06b4ddfd
W20260620 23:15:28.491420 31594 leader_election.cc:341] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525): Not found: Tablet not found: 99a6d0f8afd848d8891f15fb06b4ddfd
I20260620 23:15:28.491465 31594 leader_election.cc:302] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11482a723bfa4625a5c84d8eb153c679; no voters: 6f562397a5d54eac9ac484eaed971eb1, cce78ca247f742c18dfad6ea424477ad
I20260620 23:15:28.491554 32001 raft_consensus.cc:2726] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:15:28.583390 32002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:28.583169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:36679)
0620 23:15:28.583305 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:28.583341 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260620 23:15:28.613973 32003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:28.613720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:44521)
0620 23:15:28.613879 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:28.613918 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260620 23:15:28.747843 32004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:28.747609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:58591)
0620 23:15:28.747759 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:28.747792 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":60,"threads_started":1}
I20260620 23:15:29.584142 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:29.583911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:35599)
0620 23:15:29.584052 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:29.584097 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260620 23:15:29.614732 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:29.614489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:59815)
0620 23:15:29.614649 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:29.614688 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260620 23:15:29.748446 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:29.748243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:47481)
0620 23:15:29.748370 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:29.748408 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260620 23:15:30.585062 32008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:30.584836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:44079)
0620 23:15:30.584979 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:30.585014 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260620 23:15:30.615804 32009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:30.615523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:53961)
0620 23:15:30.615694 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:30.615740 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
I20260620 23:15:30.749230 32010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:30.748986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:42933)
0620 23:15:30.749142 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:30.749180 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260620 23:15:31.586127 32011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:31.585859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:37069)
0620 23:15:31.586033 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:31.586074 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":151,"threads_started":1}
I20260620 23:15:31.616791 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:31.616452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:36743)
0620 23:15:31.616621 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:31.616670 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260620 23:15:31.749995 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:31.749778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:50941)
0620 23:15:31.749915 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:31.749952 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260620 23:15:32.586900 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:32.586666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:38041)
0620 23:15:32.586813 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:32.586853 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
I20260620 23:15:32.617647 32015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:32.617435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:44647)
0620 23:15:32.617569 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:32.617608 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260620 23:15:32.751040 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:32.750752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:51951)
0620 23:15:32.750956 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:32.751001 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":114,"threads_started":1}
I20260620 23:15:33.587634 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:33.587417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:59643)
0620 23:15:33.587553 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:33.587586 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260620 23:15:33.618602 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:33.618292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:39079)
0620 23:15:33.618522 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:33.618562 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":183,"threads_started":1}
I20260620 23:15:33.751868 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:33.751619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:36883)
0620 23:15:33.751789 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:33.751827 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260620 23:15:33.878942 32020 raft_consensus.cc:482] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:33.879123 32020 raft_consensus.cc:504] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562397a5d54eac9ac484eaed971eb1" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 34525 } } peers { permanent_uuid: "cce78ca247f742c18dfad6ea424477ad" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 36389 } } peers { permanent_uuid: "11482a723bfa4625a5c84d8eb153c679" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 33585 } }
I20260620 23:15:33.879365 32020 leader_election.cc:288] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525), cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389)
W20260620 23:15:33.880115 31594 leader_election.cc:341] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f562397a5d54eac9ac484eaed971eb1 (127.16.79.2:34525): Not found: Tablet not found: 99a6d0f8afd848d8891f15fb06b4ddfd
W20260620 23:15:33.880198 31593 leader_election.cc:341] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cce78ca247f742c18dfad6ea424477ad (127.16.79.3:36389): Not found: Tablet not found: 99a6d0f8afd848d8891f15fb06b4ddfd
I20260620 23:15:33.880234 31593 leader_election.cc:302] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11482a723bfa4625a5c84d8eb153c679; no voters: 6f562397a5d54eac9ac484eaed971eb1, cce78ca247f742c18dfad6ea424477ad
I20260620 23:15:33.880323 32020 raft_consensus.cc:2726] T 99a6d0f8afd848d8891f15fb06b4ddfd P 11482a723bfa4625a5c84d8eb153c679 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:15:34.588625 32021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:34.588372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:58307)
0620 23:15:34.588540 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:34.588579 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260620 23:15:34.619307 32022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:34.619089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:37943)
0620 23:15:34.619237 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:34.619270 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260620 23:15:34.752748 32023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:34.752337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:51485)
0620 23:15:34.752565 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:34.752644 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":60,"threads_started":1}
I20260620 23:15:35.589612 32024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:35.589362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:36607)
0620 23:15:35.589521 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:35.589564 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260620 23:15:35.620082 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:35.619853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:60455)
0620 23:15:35.620006 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:35.620039 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260620 23:15:35.753667 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:35.753327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:48089)
0620 23:15:35.753532 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:35.753597 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":58,"threads_started":1}
I20260620 23:15:36.590376 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:36.590146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.1:44511)
0620 23:15:36.590297 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:36.590332 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260620 23:15:36.621141 32028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:36.620863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.2:52625)
0620 23:15:36.621019 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:36.621066 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260620 23:15:36.754789 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:36.754488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40353 (local address 127.16.79.3:58033)
0620 23:15:36.754692 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:36.754736 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":114,"threads_started":1}
I20260620 23:15:36.757504 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 31548
I20260620 23:15:36.761992 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 31712
I20260620 23:15:36.766897 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 31844
2026-06-20T23:15:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18313 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:15:36 dist-test-slave-pswb krb5kdc[32032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:15:36 dist-test-slave-pswb krb5kdc[32032](info): set up 2 sockets
Jun 20 23:15:36 dist-test-slave-pswb krb5kdc[32032](info): commencing operation
krb5kdc: starting...
W20260620 23:15:37.824955 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:15:37 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:15:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:15:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:15:37.953316 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45287
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:45287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:38.032739 32045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:38.032923 32045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:38.032945 32045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:38.034139 32045 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:15:38.034191 32045 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:15:38.034205 32045 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:15:38.034217 32045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:38.034232 32045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:38.034245 32045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:38.034258 32045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:38.034271 32045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:38.035701 32045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:45287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:38.035882 32045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:38.036043 32045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:38.040933 32051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:38.040958 32050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:38.041203 32045 server_base.cc:1034] running on GCE node
W20260620 23:15:38.040925 32053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:38.041812 32045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:38.042227 32045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:38.043484 32045 hybrid_clock.cc:648] HybridClock initialized: now 1781997338043459 us; error 39 us; skew 500 ppm
Jun 20 23:15:38 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:38.045967 32045 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:15:38.046347 32045 webserver.cc:460] Webserver started at http://127.16.79.62:42233/ using document root <none> and password file <none>
I20260620 23:15:38.046532 32045 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:38.046573 32045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:38.046656 32045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:38.047626 32045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "2465aef6b3ea488cbe792beddcfa9646"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "2b80e05c06bb3bd8def6afc52e15c73e"
server_key_iv: "fc66917b23b642c8aed7d3a41b87d23d"
server_key_version: "encryptionkey@0"
I20260620 23:15:38.047907 32045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "2465aef6b3ea488cbe792beddcfa9646"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "2b80e05c06bb3bd8def6afc52e15c73e"
server_key_iv: "fc66917b23b642c8aed7d3a41b87d23d"
server_key_version: "encryptionkey@0"
I20260620 23:15:38.049285 32045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:15:38.050065 32060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:38.050294 32045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:38.050395 32045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "2465aef6b3ea488cbe792beddcfa9646"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "2b80e05c06bb3bd8def6afc52e15c73e"
server_key_iv: "fc66917b23b642c8aed7d3a41b87d23d"
server_key_version: "encryptionkey@0"
I20260620 23:15:38.050468 32045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:38.079068 32045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:38.081588 32045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:38.081782 32045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:38.086632 32045 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:45287
I20260620 23:15:38.086728 32112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:45287 every 8 connection(s)
I20260620 23:15:38.087086 32045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:38.087954 32113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:38.088295 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 32045
I20260620 23:15:38.088424 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:15:38.088639 16700 external_mini_cluster.cc:1428] Setting key 01aaca762c9111f2f4dc85ef043fed14
I20260620 23:15:38.090890 32113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Bootstrap starting.
I20260620 23:15:38.091466 32113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:38.091740 32113 log.cc:824] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:38.092433 32113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: No bootstrap required, opened a new log
I20260620 23:15:38.094606 32113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } }
I20260620 23:15:38.094761 32113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:38.094811 32113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2465aef6b3ea488cbe792beddcfa9646, State: Initialized, Role: FOLLOWER
I20260620 23:15:38.094973 32113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } }
I20260620 23:15:38.095068 32113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:38.095113 32113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:38.095161 32113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 23:15:38 dist-test-slave-pswb krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:38.095712 32113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } }
I20260620 23:15:38.095834 32113 leader_election.cc:302] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2465aef6b3ea488cbe792beddcfa9646; no voters:
I20260620 23:15:38.096004 32113 leader_election.cc:288] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:15:38.096060 32118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:38.096190 32118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 1 LEADER]: Becoming Leader. State: Replica: 2465aef6b3ea488cbe792beddcfa9646, State: Running, Role: LEADER
I20260620 23:15:38.096278 32113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:38.096328 32118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } }
I20260620 23:15:38.096684 32119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2465aef6b3ea488cbe792beddcfa9646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } } }
I20260620 23:15:38.096776 32119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:38.097036 32120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2465aef6b3ea488cbe792beddcfa9646. Latest consensus state: current_term: 1 leader_uuid: "2465aef6b3ea488cbe792beddcfa9646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } } }
I20260620 23:15:38.097126 32120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: This master's current role is: LEADER
W20260620 23:15:38.098060 32134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:15:38.098501 32134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:15:38.098564 32123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:38.098747 32123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:38.099656 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.089341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.16.79.62:45287)
0620 23:15:38.089594 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:38.089605 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:38.089641 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:15:38.090215 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:38.090223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:38.090279 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:38.090586 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:38.091320 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.092300 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.093123 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.093536 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.093659 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.096254 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:38.096258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:38.096258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:38.096269 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:38.098115 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:38.098473 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:38.098475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:38.098476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:38.098592 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:38.098824 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:38.098825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:38.098825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:38.099054 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:38.099170 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:38.099344 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 23:15:38.099471 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260620 23:15:38.101078 32123 catalog_manager.cc:1347] Generated new cluster ID: 8101f7ec69164f12bc7c06c9d0eeef96
I20260620 23:15:38.101153 32123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:38.116760 32123 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:15:38.117430 32123 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:38.128026 32123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Generated new TSK 0
I20260620 23:15:38.128453 32123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:15:38.161820 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45287
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:38.250298 32139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:38.250499 32139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:38.250527 32139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:38.251786 32139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:38.251828 32139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:38.251855 32139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:15:38.253485 32139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:38.253710 32139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:38.253924 32139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:38.258999 32147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:38.259052 32145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:38.259019 32144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:38.259217 32139 server_base.cc:1034] running on GCE node
I20260620 23:15:38.259423 32139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:38.259654 32139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:38.260802 32139 hybrid_clock.cc:648] HybridClock initialized: now 1781997338260787 us; error 41 us; skew 500 ppm
Jun 20 23:15:38 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:38.263717 32139 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:15:38.264115 32139 webserver.cc:460] Webserver started at http://127.16.79.1:42211/ using document root <none> and password file <none>
I20260620 23:15:38.264361 32139 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:38.264432 32139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:38.264580 32139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:38.265748 32139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "a6844eaef5f244a48f0af6c373274c80"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "6c906e0f70bf143d5501f639df7260b8"
server_key_iv: "c17b77a601d7ca43fc0010afa1db4194"
server_key_version: "encryptionkey@0"
I20260620 23:15:38.266144 32139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "a6844eaef5f244a48f0af6c373274c80"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "6c906e0f70bf143d5501f639df7260b8"
server_key_iv: "c17b77a601d7ca43fc0010afa1db4194"
server_key_version: "encryptionkey@0"
I20260620 23:15:38.267575 32139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:15:38.268880 32154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:38.269209 32139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:38.269326 32139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "a6844eaef5f244a48f0af6c373274c80"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "6c906e0f70bf143d5501f639df7260b8"
server_key_iv: "c17b77a601d7ca43fc0010afa1db4194"
server_key_version: "encryptionkey@0"
I20260620 23:15:38.269416 32139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:38.279866 32139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:38.282100 32139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:38.282262 32139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:38.282480 32139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:38.282786 32139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:38.282842 32139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:38.282882 32139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:38.282910 32139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:38.288985 32139 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:40145
I20260620 23:15:38.289077 32267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:40145 every 8 connection(s)
I20260620 23:15:38.289366 32139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:15:38 dist-test-slave-pswb krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:38.297161 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 32139
I20260620 23:15:38.297277 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:15:38.297724 16700 external_mini_cluster.cc:1428] Setting key 46ba44255a953e177f2bdc13f5584a92
I20260620 23:15:38.299054 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.290049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:36851 (local address 127.16.79.62:45287)
0620 23:15:38.290124 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:38.290126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:38.290311 (+ 185us) server_negotiation.cc:407] Connection header received
0620 23:15:38.290753 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:38.290754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:38.290764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:38.290818 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:38.291642 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.292128 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.293253 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.293456 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.293520 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.295611 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:38.295613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:38.295613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:38.295619 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:38.297448 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:38.297985 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:38.297986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:38.297987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:38.298050 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:38.298362 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:38.298362 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:38.298363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:38.298500 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:38.298566 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:38.298906 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 23:15:38.298992 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:15:38.299309 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.290026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:36851)
0620 23:15:38.290166 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:38.290204 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:38.290664 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:38.290872 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:38.290875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:38.291385 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:38.291560 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.291565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.292237 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.292238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.293127 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.293130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.293546 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.293547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.293650 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.294174 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:38.294182 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:38.295524 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:38.297565 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:38.297573 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:38.297575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:38.297877 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:38.298138 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:38.298139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:38.298139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:38.298250 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:38.298577 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:38.298585 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:38.298746 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 23:15:38.298860 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:15:38.299029 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260620 23:15:38.300083 32268 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45287
I20260620 23:15:38.300184 32268 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:38.300355 32268 heartbeater.cc:511] Master 127.16.79.62:45287 requested a full tablet report, sending...
I20260620 23:15:38.300931 32077 ts_manager.cc:194] Registered new tserver with Master: a6844eaef5f244a48f0af6c373274c80 (127.16.79.1:40145)
I20260620 23:15:38.301533 32077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:36851
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:15:38.323041 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45287
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:38.400591 32273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:38.400790 32273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:38.400816 32273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:38.401839 32273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:38.401873 32273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:38.401898 32273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:15:38.403204 32273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:38.403364 32273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:38.403519 32273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:38.409176 32279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:38.409178 32278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:38.409221 32281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:38.409209 32273 server_base.cc:1034] running on GCE node
I20260620 23:15:38.409590 32273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:38.409832 32273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:38.411019 32273 hybrid_clock.cc:648] HybridClock initialized: now 1781997338410998 us; error 50 us; skew 500 ppm
Jun 20 23:15:38 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:38.413506 32273 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:15:38.414103 32273 webserver.cc:460] Webserver started at http://127.16.79.2:45669/ using document root <none> and password file <none>
I20260620 23:15:38.414333 32273 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:38.414403 32273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:38.414546 32273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:38.415522 32273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "d3dfef9c251a4f9aae6bf57a7caffce7"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "7c6a1d4d06384cbb4b89dd481d1ce037"
server_key_iv: "1d29afbf528a5847ad1cfdd9f20db1ff"
server_key_version: "encryptionkey@0"
I20260620 23:15:38.415872 32273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "d3dfef9c251a4f9aae6bf57a7caffce7"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "7c6a1d4d06384cbb4b89dd481d1ce037"
server_key_iv: "1d29afbf528a5847ad1cfdd9f20db1ff"
server_key_version: "encryptionkey@0"
I20260620 23:15:38.417544 32273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:15:38.418712 32288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:38.418938 32273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:38.419067 32273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "d3dfef9c251a4f9aae6bf57a7caffce7"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "7c6a1d4d06384cbb4b89dd481d1ce037"
server_key_iv: "1d29afbf528a5847ad1cfdd9f20db1ff"
server_key_version: "encryptionkey@0"
I20260620 23:15:38.419147 32273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:38.442910 32273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:38.444918 32273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:38.445062 32273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:38.445279 32273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:38.445561 32273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:38.445613 32273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:38.445653 32273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:38.445690 32273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:38.451654 32273 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:35419
I20260620 23:15:38.451699 32401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:35419 every 8 connection(s)
I20260620 23:15:38.451992 32273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:15:38.457446 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 32273
I20260620 23:15:38.457566 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:15:38.457791 16700 external_mini_cluster.cc:1428] Setting key 564037672c12669161a3f7623736ca1d
Jun 20 23:15:38 dist-test-slave-pswb krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:38.461485 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.452793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:57227 (local address 127.16.79.62:45287)
0620 23:15:38.452834 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:38.452836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:38.453033 (+ 197us) server_negotiation.cc:407] Connection header received
0620 23:15:38.453501 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:38.453502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:38.453511 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:38.453588 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:38.454483 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.454975 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.455905 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.456087 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.456150 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.458409 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:38.458410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:38.458411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:38.458418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:38.460064 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:38.460493 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:38.460494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:38.460494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:38.460557 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:38.460863 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:38.460863 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:38.460864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:38.460978 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:38.461055 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:38.461320 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 23:15:38.461398 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:15:38.461648 32404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.452763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:57227)
0620 23:15:38.452906 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:38.452945 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:38.453422 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:38.453619 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:38.453622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:38.454173 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:38.454351 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.454355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.455059 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.455060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.455818 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.455820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.456170 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.456171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.456265 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.456684 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:38.456735 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:38.458304 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:38.460154 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:38.460160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:38.460162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:38.460395 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:38.460643 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:38.460645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:38.460645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:38.460794 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:38.461046 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:38.461052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:38.461185 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 23:15:38.461284 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:15:38.461427 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260620 23:15:38.462052 32402 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45287
I20260620 23:15:38.462148 32402 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:38.462327 32402 heartbeater.cc:511] Master 127.16.79.62:45287 requested a full tablet report, sending...
I20260620 23:15:38.462663 32077 ts_manager.cc:194] Registered new tserver with Master: d3dfef9c251a4f9aae6bf57a7caffce7 (127.16.79.2:35419)
I20260620 23:15:38.463060 32077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:57227
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:15:38.487874 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45287
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:38.574446 32407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:38.574661 32407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:38.574687 32407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:38.575834 32407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:38.575875 32407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:38.575902 32407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:15:38.577406 32407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:38.577626 32407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:38.577875 32407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:38.583461 32415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:38.583834 32413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:38.583961 32412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:38.583971 32407 server_base.cc:1034] running on GCE node
I20260620 23:15:38.584479 32407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:38.584776 32407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:38.585939 32407 hybrid_clock.cc:648] HybridClock initialized: now 1781997338585929 us; error 28 us; skew 500 ppm
Jun 20 23:15:38 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:38.588892 32407 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:15:38.589537 32407 webserver.cc:460] Webserver started at http://127.16.79.3:33825/ using document root <none> and password file <none>
I20260620 23:15:38.589818 32407 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:38.589895 32407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:38.590057 32407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:38.591317 32407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "0dccc888c21a46b0bfd59a29c735a969"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "0d7a7db66307bb3442306cd5e4c81acb"
server_key_iv: "97fcf3ef59416019e80c47092d694bdc"
server_key_version: "encryptionkey@0"
I20260620 23:15:38.591722 32407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "0dccc888c21a46b0bfd59a29c735a969"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "0d7a7db66307bb3442306cd5e4c81acb"
server_key_iv: "97fcf3ef59416019e80c47092d694bdc"
server_key_version: "encryptionkey@0"
I20260620 23:15:38.593374 32407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 23:15:38.594485 32422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:38.594702 32407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:38.594812 32407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "0dccc888c21a46b0bfd59a29c735a969"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "0d7a7db66307bb3442306cd5e4c81acb"
server_key_iv: "97fcf3ef59416019e80c47092d694bdc"
server_key_version: "encryptionkey@0"
I20260620 23:15:38.594902 32407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:38.605324 32407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:38.607771 32407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:38.607929 32407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:38.608158 32407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:38.608465 32407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:38.608520 32407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:38.608559 32407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:38.608588 32407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:38.615180 32407 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:39541
I20260620 23:15:38.615221 32535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:39541 every 8 connection(s)
I20260620 23:15:38.615535 32407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:15:38 dist-test-slave-pswb krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:38.623901 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 32407
I20260620 23:15:38.624060 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:15:38.624306 16700 external_mini_cluster.cc:1428] Setting key 2750579c492d911e681a46ffcee230e1
I20260620 23:15:38.625505 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.616245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:39647 (local address 127.16.79.62:45287)
0620 23:15:38.616321 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:38.616324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:38.616501 (+ 177us) server_negotiation.cc:407] Connection header received
0620 23:15:38.617060 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:38.617062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:38.617071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:38.617141 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:38.618036 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.618536 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.619493 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.619670 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.619731 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.621804 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:38.621806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:38.621806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:38.621812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:38.623547 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:38.624006 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:38.624007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:38.624008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:38.624098 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:38.624463 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:38.624464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:38.624465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:38.624631 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:38.624762 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:38.625283 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 23:15:38.625384 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:15:38.625890 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.616203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:39647)
0620 23:15:38.616385 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:38.616420 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:38.616957 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:38.617192 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:38.617195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:38.617740 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:38.617922 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.617926 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.618638 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.618639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.619412 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.619413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.619749 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.619750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.619844 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.620293 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:38.620301 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:38.621724 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:38.623667 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:38.623673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:38.623674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:38.623904 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:38.624182 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:38.624183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:38.624183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:38.624307 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:38.624891 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:38.624903 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:38.625102 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 23:15:38.625241 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:15:38.625494 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":105,"threads_started":1}
I20260620 23:15:38.626699 32536 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45287
I20260620 23:15:38.626796 32536 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:38.627007 32536 heartbeater.cc:511] Master 127.16.79.62:45287 requested a full tablet report, sending...
I20260620 23:15:38.627478 32077 ts_manager.cc:194] Registered new tserver with Master: 0dccc888c21a46b0bfd59a29c735a969 (127.16.79.3:39541)
I20260620 23:15:38.627835 32077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:39647
I20260620 23:15:38.636723 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:15:38.644747 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.638120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.16.79.62:45287)
0620 23:15:38.638221 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:38.638223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:38.638289 (+ 66us) server_negotiation.cc:407] Connection header received
0620 23:15:38.638366 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:38.638367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:38.638378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:38.638437 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:38.639031 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.639505 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.640209 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.640387 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.640442 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.641823 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:38.641824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:38.641825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:38.641830 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:38.643353 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:38.643711 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:38.643711 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:38.643712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:38.643761 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:38.643948 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:38.643949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:38.643949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:38.644094 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:38.644184 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:38.644460 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 23:15:38.644607 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:15:38.646665 32077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:15:38.648300 32077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:15:38.655006 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.650882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.16.79.3:39541)
0620 23:15:38.651237 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:38.651241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:38.651249 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:15:38.651285 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:38.651287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:38.651303 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:38.651371 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:38.651556 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.652587 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.653871 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.654592 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.654669 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.654729 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:38.654886 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 23:15:38.654921 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":83,"threads_started":1}
I20260620 23:15:38.655009 32547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.650643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:40145 (local address 127.0.0.1:55304)
0620 23:15:38.650800 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:38.650836 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:38.650901 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:38.651184 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:38.651187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:38.651189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:38.651242 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.651242 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.652528 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.652529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.653926 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.653929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.654752 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.654753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.654816 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.654819 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:15:38.654871 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 23:15:38.654911 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":31,"thread_start_us":86,"threads_started":1}
I20260620 23:15:38.655010 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.650643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:39541 (local address 127.0.0.1:49992)
0620 23:15:38.651038 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:38.651048 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:38.651123 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:38.651437 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:38.651438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:38.651440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:38.651485 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.651486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.652688 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.652705 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.653788 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.653790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.654692 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.654693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.654770 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.654783 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:15:38.654856 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:15:38.654900 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":6,"thread_start_us":61,"threads_started":1}
I20260620 23:15:38.655339 32550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.650823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.16.79.1:40145)
0620 23:15:38.651003 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:38.651009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:38.651021 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:15:38.651064 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:38.651066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:38.651081 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:38.651166 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:38.651310 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.652436 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.653993 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.654660 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.655128 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.655203 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:38.655231 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 23:15:38.655270 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260620 23:15:38.656040 32202 tablet_service.cc:1467] Processing CreateTablet for tablet b45a7785420641398b46be0a720f167d (DEFAULT_TABLE table=test-table [id=51690e977686457bbea05f55ceac892e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:38.656111 32470 tablet_service.cc:1467] Processing CreateTablet for tablet b45a7785420641398b46be0a720f167d (DEFAULT_TABLE table=test-table [id=51690e977686457bbea05f55ceac892e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:38.656412 32470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b45a7785420641398b46be0a720f167d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:38.656411 32202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b45a7785420641398b46be0a720f167d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:38.656590 32548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.650657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:35419 (local address 127.0.0.1:39028)
0620 23:15:38.652747 (+ 2090us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:38.652757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:38.652807 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:38.653006 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:38.653007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:38.653009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:38.653050 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.653051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.654273 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.654274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.655289 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.655291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.656342 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.656343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.656420 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.656424 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:15:38.656475 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 23:15:38.656526 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2072,"thread_start_us":87,"threads_started":1}
I20260620 23:15:38.656889 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.650895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.16.79.2:35419)
0620 23:15:38.651046 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:38.651050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:38.652847 (+ 1797us) server_negotiation.cc:407] Connection header received
0620 23:15:38.652895 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:38.652898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:38.652912 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:38.652994 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:38.653137 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.654180 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.655368 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.656179 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.656255 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.656305 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:38.656761 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 23:15:38.656810 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260620 23:15:38.657450 32336 tablet_service.cc:1467] Processing CreateTablet for tablet b45a7785420641398b46be0a720f167d (DEFAULT_TABLE table=test-table [id=51690e977686457bbea05f55ceac892e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:38.657768 32336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b45a7785420641398b46be0a720f167d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:38.659161 32553 tablet_bootstrap.cc:492] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Bootstrap starting.
I20260620 23:15:38.659749 32554 tablet_bootstrap.cc:492] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969: Bootstrap starting.
I20260620 23:15:38.659984 32553 tablet_bootstrap.cc:654] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:38.660368 32554 tablet_bootstrap.cc:654] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:38.660519 32553 log.cc:824] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:38.661000 32554 log.cc:824] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:38.661296 32553 tablet_bootstrap.cc:492] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: No bootstrap required, opened a new log
I20260620 23:15:38.661307 32556 tablet_bootstrap.cc:492] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7: Bootstrap starting.
I20260620 23:15:38.661375 32553 ts_tablet_manager.cc:1397] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:15:38.661707 32554 tablet_bootstrap.cc:492] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969: No bootstrap required, opened a new log
I20260620 23:15:38.661777 32554 ts_tablet_manager.cc:1397] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:15:38.661921 32556 tablet_bootstrap.cc:654] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:38.662395 32556 log.cc:824] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:38.662969 32556 tablet_bootstrap.cc:492] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7: No bootstrap required, opened a new log
I20260620 23:15:38.663008 32556 ts_tablet_manager.cc:1397] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:15:38.663589 32553 raft_consensus.cc:348] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } }
I20260620 23:15:38.663784 32553 raft_consensus.cc:374] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:38.663821 32553 raft_consensus.cc:729] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6844eaef5f244a48f0af6c373274c80, State: Initialized, Role: FOLLOWER
I20260620 23:15:38.663945 32553 consensus_queue.cc:260] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } }
I20260620 23:15:38.663919 32554 raft_consensus.cc:348] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } }
I20260620 23:15:38.664095 32554 raft_consensus.cc:374] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:38.664135 32554 raft_consensus.cc:729] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dccc888c21a46b0bfd59a29c735a969, State: Initialized, Role: FOLLOWER
I20260620 23:15:38.664253 32268 heartbeater.cc:503] Master 127.16.79.62:45287 was elected leader, sending a full tablet report...
I20260620 23:15:38.664238 32553 ts_tablet_manager.cc:1428] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:15:38.664580 32554 consensus_queue.cc:260] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } }
I20260620 23:15:38.664956 32554 ts_tablet_manager.cc:1428] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 23:15:38.664968 32536 heartbeater.cc:503] Master 127.16.79.62:45287 was elected leader, sending a full tablet report...
I20260620 23:15:38.665294 32556 raft_consensus.cc:348] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } }
I20260620 23:15:38.665482 32556 raft_consensus.cc:374] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:38.665566 32556 raft_consensus.cc:729] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3dfef9c251a4f9aae6bf57a7caffce7, State: Initialized, Role: FOLLOWER
I20260620 23:15:38.665740 32556 consensus_queue.cc:260] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } }
I20260620 23:15:38.666023 32556 ts_tablet_manager.cc:1428] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:15:38.666092 32402 heartbeater.cc:503] Master 127.16.79.62:45287 was elected leader, sending a full tablet report...
W20260620 23:15:38.702739 32403 tablet.cc:2369] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:38.703543 32559 raft_consensus.cc:482] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:38.703676 32559 raft_consensus.cc:504] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } }
I20260620 23:15:38.704057 32559 leader_election.cc:288] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dccc888c21a46b0bfd59a29c735a969 (127.16.79.3:39541), d3dfef9c251a4f9aae6bf57a7caffce7 (127.16.79.2:35419)
I20260620 23:15:38.706818 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.704295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:42989 (local address 127.16.79.3:39541)
0620 23:15:38.704375 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:38.704377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:38.704387 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:15:38.704414 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:38.704415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:38.704426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:38.704478 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:38.704641 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.705161 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.706082 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.706465 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.706548 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.706559 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:38.706719 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 23:15:38.706756 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:15:38.706818 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.704231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:39541 (local address 127.16.79.1:42989)
0620 23:15:38.704296 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:38.704308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:38.704350 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:38.704481 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:38.704481 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:38.704483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:38.704562 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.704563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.705248 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.705249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.705982 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.705983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.706548 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.706549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.706632 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.706635 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:15:38.706710 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:15:38.706752 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:15:38.706952 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.704310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:49357 (local address 127.16.79.2:35419)
0620 23:15:38.704366 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:38.704368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:38.704518 (+ 150us) server_negotiation.cc:407] Connection header received
0620 23:15:38.704603 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:38.704604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:38.704616 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:38.704686 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:38.704859 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.705320 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.706222 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.706591 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.706682 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.706693 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:38.706844 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 23:15:38.706881 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:15:38.706992 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.704240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:35419 (local address 127.16.79.1:49357)
0620 23:15:38.704441 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:38.704449 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:38.704531 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:38.704710 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:38.704710 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:38.704712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:38.704787 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.704789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.705401 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.705402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.706131 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:38.706132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.706684 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.706685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:38.706761 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.706765 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:15:38.706826 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:15:38.706865 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":4,"thread_start_us":109,"threads_started":1}
I20260620 23:15:38.707242 32490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45a7785420641398b46be0a720f167d" candidate_uuid: "a6844eaef5f244a48f0af6c373274c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dccc888c21a46b0bfd59a29c735a969" is_pre_election: true
I20260620 23:15:38.707307 32356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45a7785420641398b46be0a720f167d" candidate_uuid: "a6844eaef5f244a48f0af6c373274c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" is_pre_election: true
I20260620 23:15:38.707397 32490 raft_consensus.cc:2454] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6844eaef5f244a48f0af6c373274c80 in term 0.
I20260620 23:15:38.707413 32356 raft_consensus.cc:2454] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6844eaef5f244a48f0af6c373274c80 in term 0.
I20260620 23:15:38.707640 32157 leader_election.cc:302] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dccc888c21a46b0bfd59a29c735a969, a6844eaef5f244a48f0af6c373274c80; no voters:
I20260620 23:15:38.707763 32559 raft_consensus.cc:2781] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:15:38.707852 32559 raft_consensus.cc:482] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:15:38.707885 32559 raft_consensus.cc:3037] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:38.708590 32559 raft_consensus.cc:504] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } }
I20260620 23:15:38.708766 32559 leader_election.cc:288] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [CANDIDATE]: Term 1 election: Requested vote from peers 0dccc888c21a46b0bfd59a29c735a969 (127.16.79.3:39541), d3dfef9c251a4f9aae6bf57a7caffce7 (127.16.79.2:35419)
I20260620 23:15:38.708894 32490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45a7785420641398b46be0a720f167d" candidate_uuid: "a6844eaef5f244a48f0af6c373274c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dccc888c21a46b0bfd59a29c735a969"
I20260620 23:15:38.708930 32356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45a7785420641398b46be0a720f167d" candidate_uuid: "a6844eaef5f244a48f0af6c373274c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7"
I20260620 23:15:38.708969 32490 raft_consensus.cc:3037] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:38.709007 32356 raft_consensus.cc:3037] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:38.709671 32356 raft_consensus.cc:2454] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6844eaef5f244a48f0af6c373274c80 in term 1.
I20260620 23:15:38.709746 32490 raft_consensus.cc:2454] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6844eaef5f244a48f0af6c373274c80 in term 1.
I20260620 23:15:38.709923 32155 leader_election.cc:302] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6844eaef5f244a48f0af6c373274c80, d3dfef9c251a4f9aae6bf57a7caffce7; no voters:
I20260620 23:15:38.710038 32559 raft_consensus.cc:2781] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:38.710209 32559 raft_consensus.cc:686] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 1 LEADER]: Becoming Leader. State: Replica: a6844eaef5f244a48f0af6c373274c80, State: Running, Role: LEADER
I20260620 23:15:38.710302 32559 consensus_queue.cc:237] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } }
I20260620 23:15:38.711084 32076 catalog_manager.cc:5495] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6844eaef5f244a48f0af6c373274c80 (127.16.79.1). New cstate: current_term: 1 leader_uuid: "a6844eaef5f244a48f0af6c373274c80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } health_report { overall_health: UNKNOWN } } }
I20260620 23:15:38.720202 32550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.717977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.16.79.1:40145)
0620 23:15:38.718028 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:38.718029 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:38.718036 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:15:38.718060 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:38.718061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:38.718076 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:38.718155 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:38.718296 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.718747 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.719473 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:38.719659 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:38.719733 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:38.719894 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:38.719978 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:15:38.720067 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:38.720086 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 23:15:38.720106 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 23:15:38.721793 32356 raft_consensus.cc:1261] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7 [term 1 FOLLOWER]: Refusing update from remote peer a6844eaef5f244a48f0af6c373274c80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:38.722059 32490 raft_consensus.cc:1261] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969 [term 1 FOLLOWER]: Refusing update from remote peer a6844eaef5f244a48f0af6c373274c80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:38.722086 32559 consensus_queue.cc:1035] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:38.722414 32559 consensus_queue.cc:1035] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:38.724581 32566 mvcc.cc:205] Tried to move back new op lower bound from 7299061099402219520 to 7299061099358076928. Current Snapshot: MvccSnapshot[applied={T|T < 7299061099402219520}]
I20260620 23:15:38.725401 32567 mvcc.cc:205] Tried to move back new op lower bound from 7299061099402219520 to 7299061099358076928. Current Snapshot: MvccSnapshot[applied={T|T < 7299061099402219520}]
I20260620 23:15:38.727394 32077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:38.727499 32077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:38.729605 32077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Sending DeleteTablet for 3 replicas of tablet b45a7785420641398b46be0a720f167d
I20260620 23:15:38.729998 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 32139
I20260620 23:15:38.730032 32202 tablet_service.cc:1514] Processing DeleteTablet for tablet b45a7785420641398b46be0a720f167d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:38 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:55304
I20260620 23:15:38.730121 32336 tablet_service.cc:1514] Processing DeleteTablet for tablet b45a7785420641398b46be0a720f167d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:38 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:39028
I20260620 23:15:38.730659 32575 tablet_replica.cc:330] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7: stopping tablet replica
I20260620 23:15:38.730775 32575 raft_consensus.cc:2229] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:38.730888 32575 raft_consensus.cc:2258] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:38.732203 32470 tablet_service.cc:1514] Processing DeleteTablet for tablet b45a7785420641398b46be0a720f167d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:38 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:49992
I20260620 23:15:38.732565 32576 tablet_replica.cc:330] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969: stopping tablet replica
I20260620 23:15:38.732571 32575 ts_tablet_manager.cc:1905] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:38.732684 32576 raft_consensus.cc:2229] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:38.732825 32576 raft_consensus.cc:2258] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:38.733253 32576 ts_tablet_manager.cc:1905] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:38.734805 32576 ts_tablet_manager.cc:1918] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:38.734891 32576 log.cc:1196] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/b45a7785420641398b46be0a720f167d
I20260620 23:15:38.735136 32575 ts_tablet_manager.cc:1918] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:38.735168 32576 ts_tablet_manager.cc:1939] T b45a7785420641398b46be0a720f167d P 0dccc888c21a46b0bfd59a29c735a969: Deleting consensus metadata
I20260620 23:15:38.735256 32575 log.cc:1196] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/b45a7785420641398b46be0a720f167d
I20260620 23:15:38.735563 32575 ts_tablet_manager.cc:1939] T b45a7785420641398b46be0a720f167d P d3dfef9c251a4f9aae6bf57a7caffce7: Deleting consensus metadata
I20260620 23:15:38.735549 32063 catalog_manager.cc:4841] TS 0dccc888c21a46b0bfd59a29c735a969 (127.16.79.3:39541): tablet b45a7785420641398b46be0a720f167d (table test-table [id=51690e977686457bbea05f55ceac892e]) successfully deleted
I20260620 23:15:38.735998 32061 catalog_manager.cc:4841] TS d3dfef9c251a4f9aae6bf57a7caffce7 (127.16.79.2:35419): tablet b45a7785420641398b46be0a720f167d (table test-table [id=51690e977686457bbea05f55ceac892e]) successfully deleted
W20260620 23:15:38.736956 32062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:40145 (error 108)
I20260620 23:15:38.737488 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 32273
I20260620 23:15:38.737730 32548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:38.737521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:40145 (local address 127.0.0.1:55320)
0620 23:15:38.737626 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:38.737680 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 23:15:38.738221 32062 catalog_manager.cc:4584] TS a6844eaef5f244a48f0af6c373274c80 (127.16.79.1:40145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b45a7785420641398b46be0a720f167d: Network error: Client connection negotiation failed: client connection to 127.16.79.1:40145: connect: Connection refused (error 111)
I20260620 23:15:38.742864 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 32407
I20260620 23:15:38.747151 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 32045
I20260620 23:15:58.752384 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45287
--webserver_interface=127.16.79.62
--webserver_port=42233
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:45287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:58.838476 32578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:58.838670 32578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:58.838692 32578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:58.839805 32578 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:15:58.839844 32578 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:15:58.839857 32578 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:15:58.839869 32578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:58.839881 32578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:58.839895 32578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:58.839944 32578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:58.839967 32578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:58.841462 32578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:45287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=42233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:58.841703 32578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:58.841945 32578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:15:58.853700 32578 server_base.cc:1034] running on GCE node
W20260620 23:15:58.853657 32585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:58.853662 32587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:58.853642 32584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:58.854230 32578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:58.854609 32578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:58.855788 32578 hybrid_clock.cc:648] HybridClock initialized: now 1781997358855779 us; error 53 us; skew 500 ppm
Jun 20 23:15:58 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997358, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:58.858951 32578 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:15:58.859547 32578 webserver.cc:460] Webserver started at http://127.16.79.62:42233/ using document root <none> and password file <none>
I20260620 23:15:58.859845 32578 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:58.859915 32578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:58.861666 32578 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:58.862578 32594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:58.862802 32578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:58.862893 32578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "2465aef6b3ea488cbe792beddcfa9646"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "2b80e05c06bb3bd8def6afc52e15c73e"
server_key_iv: "fc66917b23b642c8aed7d3a41b87d23d"
server_key_version: "encryptionkey@0"
I20260620 23:15:58.863339 32578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:58.888298 32578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:58.890621 32578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:58.890733 32578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:58.894608 32578 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:45287
I20260620 23:15:58.894688 32646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:45287 every 8 connection(s)
I20260620 23:15:58.895074 32578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:58.896618 32647 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:15:58.897313 32647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Bootstrap starting.
I20260620 23:15:58.898016 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 32578
I20260620 23:15:58.898362 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:40145
--local_ip_for_outbound_sockets=127.16.79.1
--tserver_master_addrs=127.16.79.62:45287
--webserver_port=42211
--webserver_interface=127.16.79.1
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:58.906494 32647 log.cc:824] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:58.908507 32647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:15:58.908874 32647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Bootstrap complete.
I20260620 23:15:58.911079 32647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } }
I20260620 23:15:58.911330 32647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2465aef6b3ea488cbe792beddcfa9646, State: Initialized, Role: FOLLOWER
I20260620 23:15:58.911511 32647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } }
I20260620 23:15:58.911602 32647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:58.911659 32647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:58.911706 32647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:15:58.912834 32647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } }
I20260620 23:15:58.912946 32647 leader_election.cc:302] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2465aef6b3ea488cbe792beddcfa9646; no voters:
I20260620 23:15:58.913117 32647 leader_election.cc:288] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:15:58.913157 32651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:15:58.913321 32651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 2 LEADER]: Becoming Leader. State: Replica: 2465aef6b3ea488cbe792beddcfa9646, State: Running, Role: LEADER
I20260620 23:15:58.913380 32647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:58.914330 32651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } }
I20260620 23:15:58.914909 32653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2465aef6b3ea488cbe792beddcfa9646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } } }
I20260620 23:15:58.915143 32653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:58.915055 32652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2465aef6b3ea488cbe792beddcfa9646. Latest consensus state: current_term: 2 leader_uuid: "2465aef6b3ea488cbe792beddcfa9646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } } }
I20260620 23:15:58.915326 32652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:58.915529 32665 catalog_manager.cc:1259] Loaded cluster ID: 8101f7ec69164f12bc7c06c9d0eeef96
I20260620 23:15:58.915576 32665 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: loading cluster ID for follower catalog manager: success
I20260620 23:15:58.916759 32665 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: acquiring CA information for follower catalog manager: success
I20260620 23:15:58.917147 32665 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:15:58.917225 32668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:58.917622 32668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:58.917717 32668 catalog_manager.cc:1259] Loaded cluster ID: 8101f7ec69164f12bc7c06c9d0eeef96
I20260620 23:15:58.917900 32668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:58.918370 32668 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:58.918546 32668 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Loaded TSK: 0
I20260620 23:15:58.929244 32668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Generated new TSK 1
I20260620 23:15:58.929334 32668 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:15:58.982409 32649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:58.982712 32649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:58.982757 32649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:58.984117 32649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:58.984190 32649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:58.984243 32649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:15:58.985870 32649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:40145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=42211
--tserver_master_addrs=127.16.79.62:45287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:58.986171 32649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:58.986472 32649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:58.992215 32675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:58.992215 32674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:58.992416 32677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:58.992949 32649 server_base.cc:1034] running on GCE node
I20260620 23:15:58.993196 32649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:58.993572 32649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:58.994815 32649 hybrid_clock.cc:648] HybridClock initialized: now 1781997358994798 us; error 34 us; skew 500 ppm
Jun 20 23:15:58 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997358, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:58.997741 32649 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:15:58.998368 32649 webserver.cc:460] Webserver started at http://127.16.79.1:42211/ using document root <none> and password file <none>
I20260620 23:15:58.998642 32649 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:58.998732 32649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:59.000413 32649 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:59.001461 32684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:59.001721 32649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:59.001830 32649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "a6844eaef5f244a48f0af6c373274c80"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "6c906e0f70bf143d5501f639df7260b8"
server_key_iv: "c17b77a601d7ca43fc0010afa1db4194"
server_key_version: "encryptionkey@0"
I20260620 23:15:59.002251 32649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:59.008641 32649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:59.010860 32649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:59.011014 32649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:59.011255 32649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:59.011678 32691 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 23:15:59.013698 32649 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:15:59.013749 32649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 23:15:59.013784 32649 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:15:59.014624 32649 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:15:59.014667 32649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 23:15:59.014770 32691 tablet_bootstrap.cc:492] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Bootstrap starting.
I20260620 23:15:59.022965 32649 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:40145
I20260620 23:15:59.023140 330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:40145 every 8 connection(s)
I20260620 23:15:59.023633 32649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:15:59.025911 32691 log.cc:824] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:59.028559 32691 tablet_bootstrap.cc:492] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:15:59.028923 32691 tablet_bootstrap.cc:492] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Bootstrap complete.
I20260620 23:15:59.030094 32691 ts_tablet_manager.cc:1397] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260620 23:15:59.031450 32691 raft_consensus.cc:348] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } }
I20260620 23:15:59.031790 32691 raft_consensus.cc:729] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6844eaef5f244a48f0af6c373274c80, State: Initialized, Role: FOLLOWER
I20260620 23:15:59.031962 32691 consensus_queue.cc:260] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } }
I20260620 23:15:59.032243 32691 ts_tablet_manager.cc:1428] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
Jun 20 23:15:59 dist-test-slave-pswb krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997358, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:59.033759 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 32649
I20260620 23:15:59.034060 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:35419
--local_ip_for_outbound_sockets=127.16.79.2
--tserver_master_addrs=127.16.79.62:45287
--webserver_port=45669
--webserver_interface=127.16.79.2
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:59.037225 333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.024756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:39689)
0620 23:15:59.026401 (+ 1645us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:59.026445 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:59.027007 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:59.027441 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:59.027445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:59.027753 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:59.027938 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.027942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.029223 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.029224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.030011 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.030012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.030662 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.030663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.030799 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.031350 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:59.031358 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:59.032971 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:59.035271 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:59.035278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:59.035280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:59.035555 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:59.035873 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:59.035874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:59.035874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:59.036005 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:59.036496 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:59.036549 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:59.036726 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 23:15:59.036846 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 23:15:59.037022 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1597,"thread_start_us":116,"threads_started":1}
I20260620 23:15:59.038043 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.024896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:39689 (local address 127.16.79.62:45287)
0620 23:15:59.025150 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:59.025160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:59.026601 (+ 1441us) server_negotiation.cc:407] Connection header received
0620 23:15:59.027229 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:59.027234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:59.027277 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:59.027411 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:59.028050 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.029106 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.030115 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.030553 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.030679 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.033076 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:59.033080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:59.033080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:59.033091 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:59.035170 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:59.035650 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:59.035654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:59.035655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:59.035786 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:59.036092 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:59.036093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:59.036093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:59.036409 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:59.036494 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:59.036926 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 23:15:59.037073 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":134,"threads_started":1}
I20260620 23:15:59.038897 331 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45287
I20260620 23:15:59.039004 331 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:59.039186 331 heartbeater.cc:511] Master 127.16.79.62:45287 requested a full tablet report, sending...
I20260620 23:15:59.040179 32611 ts_manager.cc:194] Registered new tserver with Master: a6844eaef5f244a48f0af6c373274c80 (127.16.79.1:40145)
I20260620 23:15:59.041828 32611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:39689
I20260620 23:15:59.043077 331 heartbeater.cc:503] Master 127.16.79.62:45287 was elected leader, sending a full tablet report...
I20260620 23:15:59.047736 338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.041051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:40145 (local address 127.0.0.1:60668)
0620 23:15:59.042634 (+ 1583us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:59.042683 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:59.042788 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:59.043346 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:59.043351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:59.043354 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:59.043475 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.043478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.045026 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.045028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.046327 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.046330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.047388 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.047389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.047481 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.047495 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:15:59.047576 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:15:59.047639 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1539,"thread_start_us":118,"threads_started":1}
I20260620 23:15:59.047744 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.042223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.16.79.1:40145)
0620 23:15:59.042461 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:59.042466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:59.042947 (+ 481us) server_negotiation.cc:407] Connection header received
0620 23:15:59.042998 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:59.043001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:59.043024 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:59.043107 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:59.043655 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.044910 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.046440 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.047264 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.047351 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.047414 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:59.047608 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 23:15:59.047656 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":138,"threads_started":1}
I20260620 23:15:59.048532 32733 tablet_service.cc:1514] Processing DeleteTablet for tablet b45a7785420641398b46be0a720f167d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:38 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:60668
I20260620 23:15:59.049460 340 tablet_replica.cc:330] stopping tablet replica
I20260620 23:15:59.049659 340 raft_consensus.cc:2229] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:59.049801 340 raft_consensus.cc:2258] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:59.050199 340 ts_tablet_manager.cc:1905] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:59.052805 340 ts_tablet_manager.cc:1918] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:59.052928 340 log.cc:1196] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal/wals/b45a7785420641398b46be0a720f167d
I20260620 23:15:59.053215 340 ts_tablet_manager.cc:1939] T b45a7785420641398b46be0a720f167d P a6844eaef5f244a48f0af6c373274c80: Deleting consensus metadata
I20260620 23:15:59.054337 32596 catalog_manager.cc:4841] TS a6844eaef5f244a48f0af6c373274c80 (127.16.79.1:40145): tablet b45a7785420641398b46be0a720f167d (table test-table [id=51690e977686457bbea05f55ceac892e]) successfully deleted
W20260620 23:15:59.122310 337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:59.122504 337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:59.122527 337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:59.123557 337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:59.123596 337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:59.123621 337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:15:59.124959 337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:35419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=45669
--tserver_master_addrs=127.16.79.62:45287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:59.125137 337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:59.125298 337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:59.130089 348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:59.130484 346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:59.130510 345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:59.130828 337 server_base.cc:1034] running on GCE node
I20260620 23:15:59.131007 337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:59.131217 337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:59.132454 337 hybrid_clock.cc:648] HybridClock initialized: now 1781997359132427 us; error 28 us; skew 500 ppm
Jun 20 23:15:59 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:59.138906 337 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:15:59.139429 337 webserver.cc:460] Webserver started at http://127.16.79.2:45669/ using document root <none> and password file <none>
I20260620 23:15:59.139672 337 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:59.139753 337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:59.141376 337 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:59.142311 355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:59.142606 337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:15:59.142738 337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "d3dfef9c251a4f9aae6bf57a7caffce7"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "7c6a1d4d06384cbb4b89dd481d1ce037"
server_key_iv: "1d29afbf528a5847ad1cfdd9f20db1ff"
server_key_version: "encryptionkey@0"
I20260620 23:15:59.143134 337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:59.164012 337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:59.166373 337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:59.166527 337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:59.166793 337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:59.167117 337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:59.167173 337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:59.167215 337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:59.167243 337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:59.174978 337 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:35419
I20260620 23:15:59.175040 468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:35419 every 8 connection(s)
I20260620 23:15:59.175400 337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:15:59.180938 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 337
I20260620 23:15:59.181234 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:39541
--local_ip_for_outbound_sockets=127.16.79.3
--tserver_master_addrs=127.16.79.62:45287
--webserver_port=33825
--webserver_interface=127.16.79.3
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:15:59 dist-test-slave-pswb krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:59.186296 471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.176322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:42835)
0620 23:15:59.176536 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:59.176635 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:59.177232 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:59.177576 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:59.177580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:59.178184 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:59.178386 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.178391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.179150 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.179151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.180016 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.180018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.180436 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.180437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.180568 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.181167 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:59.181175 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:59.182733 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:59.184781 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:59.184787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:59.184789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:59.185001 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:59.185259 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:59.185260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:59.185260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:59.185351 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:59.185753 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:59.185761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:59.185916 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 23:15:59.186010 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:15:59.186130 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260620 23:15:59.186365 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.176348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:42835 (local address 127.16.79.62:45287)
0620 23:15:59.176418 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:59.176420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:59.176772 (+ 352us) server_negotiation.cc:407] Connection header received
0620 23:15:59.177349 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:59.177350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:59.177363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:59.177478 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:59.178493 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.179039 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.180117 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.180344 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.180429 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.182850 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:59.182852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:59.182852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:59.182860 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:59.184601 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:59.185099 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:59.185100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:59.185101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:59.185181 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:59.185439 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:59.185440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:59.185441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:59.185616 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:59.185694 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:59.186090 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 23:15:59.186211 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:15:59.187228 469 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45287
I20260620 23:15:59.187330 469 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:59.187512 469 heartbeater.cc:511] Master 127.16.79.62:45287 requested a full tablet report, sending...
I20260620 23:15:59.188197 32611 ts_manager.cc:194] Registered new tserver with Master: d3dfef9c251a4f9aae6bf57a7caffce7 (127.16.79.2:35419)
I20260620 23:15:59.188792 32611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:42835
W20260620 23:15:59.266029 472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:59.266232 472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:59.266255 472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:59.267419 472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:59.267458 472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:59.267524 472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:15:59.269012 472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:39541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=33825
--tserver_master_addrs=127.16.79.62:45287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:15:59.269193 472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:59.269351 472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:59.274060 480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:59.274091 477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:59.274230 478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:59.274461 472 server_base.cc:1034] running on GCE node
I20260620 23:15:59.274760 472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:59.275094 472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:59.276309 472 hybrid_clock.cc:648] HybridClock initialized: now 1781997359276271 us; error 49 us; skew 500 ppm
Jun 20 23:15:59 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:59.279189 472 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:15:59.279608 472 webserver.cc:460] Webserver started at http://127.16.79.3:33825/ using document root <none> and password file <none>
I20260620 23:15:59.279856 472 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:59.279939 472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:59.281430 472 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:59.282085 487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:59.282356 472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:15:59.282511 472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "0dccc888c21a46b0bfd59a29c735a969"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "0d7a7db66307bb3442306cd5e4c81acb"
server_key_iv: "97fcf3ef59416019e80c47092d694bdc"
server_key_version: "encryptionkey@0"
I20260620 23:15:59.282967 472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:59.298926 472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:59.301290 472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:59.301450 472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:59.301679 472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:59.302014 472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:59.302078 472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:59.302120 472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:59.302148 472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:59.307905 472 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:39541
I20260620 23:15:59.308049 600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:39541 every 8 connection(s)
I20260620 23:15:59.308307 472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:15:59 dist-test-slave-pswb krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:15:59.317905 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.309105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:42153 (local address 127.16.79.62:45287)
0620 23:15:59.309152 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:59.309154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:59.309333 (+ 179us) server_negotiation.cc:407] Connection header received
0620 23:15:59.309773 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:59.309775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:59.309784 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:59.309831 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:59.310678 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.311148 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.312109 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.312306 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.312363 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.314509 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:59.314510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:59.314511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:59.314517 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:59.315977 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:59.316879 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:59.316881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:59.316881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:59.316963 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:59.317287 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:59.317289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:59.317289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:59.317422 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:59.317490 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:59.317738 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 23:15:59.317840 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:15:59.318058 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.309006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:42153)
0620 23:15:59.309217 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:59.309249 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:59.309698 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:59.309886 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:59.309889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:59.310414 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:59.310590 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.310595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.311245 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.311246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.312027 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.312029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.312396 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.312397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.312496 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.312963 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:59.312993 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:59.314429 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:59.316214 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:59.316227 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:59.316229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:59.316689 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:59.317079 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:59.317080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:59.317080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:59.317186 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:59.317498 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:59.317502 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:59.317613 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 23:15:59.317693 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:15:59.317802 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260620 23:15:59.318051 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 472
I20260620 23:15:59.318611 601 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45287
I20260620 23:15:59.318720 601 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:59.318902 601 heartbeater.cc:511] Master 127.16.79.62:45287 requested a full tablet report, sending...
I20260620 23:15:59.319373 32611 ts_manager.cc:194] Registered new tserver with Master: 0dccc888c21a46b0bfd59a29c735a969 (127.16.79.3:39541)
I20260620 23:15:59.319769 32611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:42153
I20260620 23:15:59.325341 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.318441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.16.79.62:45287)
0620 23:15:59.318481 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:59.318482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:59.318700 (+ 218us) server_negotiation.cc:407] Connection header received
0620 23:15:59.318744 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:59.318745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:59.318755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:59.318805 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:59.319571 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.320151 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.321045 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.321216 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.321280 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.322629 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:59.322630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:59.322630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:59.322636 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:59.323972 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:59.324328 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:59.324329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:59.324329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:59.324374 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:59.324592 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:59.324593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:59.324594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:59.324858 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:59.325091 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:59.325224 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 23:15:59.325284 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:15:59.326375 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:15:59.329097 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.326751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.16.79.62:45287)
0620 23:15:59.326797 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:59.326799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:59.326887 (+ 88us) server_negotiation.cc:407] Connection header received
0620 23:15:59.326972 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:59.326973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:59.327001 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:59.327088 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:59.327205 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.327664 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.328425 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.328632 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.328737 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.328858 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:59.328901 (+ 43us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 23:15:59.329037 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59948: token expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 23:15:59.329147 334 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59948: token expired
W20260620 23:15:59.329260 32542 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.79.62:45287)
I20260620 23:15:59.335561 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.329493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.16.79.62:45287)
0620 23:15:59.329555 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:59.329557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:59.329584 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:15:59.329603 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:59.329604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:59.329613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:59.329653 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:59.330182 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.330618 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.331292 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.331485 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.331558 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.332895 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:59.332897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:59.332897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:59.332904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:59.334349 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:59.334740 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:59.334741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:59.334741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:59.334803 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:59.335022 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:59.335023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:59.335023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:59.335160 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:59.335272 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:59.335389 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 23:15:59.335465 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:15:59.336656 32611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:15:59.338397 32611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:15:59.340510 32733 tablet_service.cc:1467] Processing CreateTablet for tablet 59d843c57c6a4340b0dc32a372f4d6e5 (DEFAULT_TABLE table=test-table [id=8209f9f726ae413d887f9fccb151cee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:59.340746 32733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d843c57c6a4340b0dc32a372f4d6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:59.342548 32691 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: Bootstrap starting.
I20260620 23:15:59.343132 32691 tablet_bootstrap.cc:654] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:59.343956 32691 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: No bootstrap required, opened a new log
I20260620 23:15:59.344027 32691 ts_tablet_manager.cc:1397] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 23:15:59.344199 338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.340350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:35419 (local address 127.0.0.1:38178)
0620 23:15:59.340413 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:59.340423 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:59.340460 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:59.340938 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:59.340938 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:59.340939 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:59.340973 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.340974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.342096 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.342097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.342859 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.342861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.343865 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.343866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.343965 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.343972 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:15:59.344044 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 23:15:59.344119 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:15:59.344177 32691 raft_consensus.cc:348] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:15:59.344272 32691 raft_consensus.cc:374] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:59.344357 32691 raft_consensus.cc:729] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6844eaef5f244a48f0af6c373274c80, State: Initialized, Role: FOLLOWER
I20260620 23:15:59.344424 32691 consensus_queue.cc:260] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:15:59.344522 32691 ts_tablet_manager.cc:1428] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:59.344614 607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.340567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.16.79.2:35419)
0620 23:15:59.340776 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:59.340782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:59.340794 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:15:59.340833 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:59.340835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:59.340851 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:59.340937 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:59.341016 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.342001 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.342964 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.343771 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.343852 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.343906 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:59.344418 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 23:15:59.344499 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":95,"threads_started":1}
I20260620 23:15:59.345620 403 tablet_service.cc:1467] Processing CreateTablet for tablet 59d843c57c6a4340b0dc32a372f4d6e5 (DEFAULT_TABLE table=test-table [id=8209f9f726ae413d887f9fccb151cee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:59.346153 403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d843c57c6a4340b0dc32a372f4d6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:59.346487 608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.342653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.16.79.3:39541)
0620 23:15:59.342941 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:59.342946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:59.342957 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:15:59.342995 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:59.342996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:59.343012 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:59.343086 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:59.343413 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.344443 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.345542 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.346168 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.346238 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.346289 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:59.346376 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 23:15:59.346412 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":173,"threads_started":1}
I20260620 23:15:59.346480 606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.340350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:39541 (local address 127.0.0.1:39018)
0620 23:15:59.340757 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:59.340767 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:59.340815 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:59.343286 (+ 2471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:59.343287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:59.343291 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:59.343337 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.343339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.344538 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.344539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.345418 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.345421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.346238 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.346239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.346301 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.346305 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:15:59.346363 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 23:15:59.346407 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":21,"thread_start_us":57,"threads_started":1}
I20260620 23:15:59.347261 535 tablet_service.cc:1467] Processing CreateTablet for tablet 59d843c57c6a4340b0dc32a372f4d6e5 (DEFAULT_TABLE table=test-table [id=8209f9f726ae413d887f9fccb151cee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:59.347585 535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d843c57c6a4340b0dc32a372f4d6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:59.349731 610 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Bootstrap starting.
I20260620 23:15:59.350292 610 tablet_bootstrap.cc:654] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:59.350754 611 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: Bootstrap starting.
I20260620 23:15:59.350775 610 log.cc:824] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:59.351341 611 tablet_bootstrap.cc:654] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:59.351593 610 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: No bootstrap required, opened a new log
I20260620 23:15:59.351691 610 ts_tablet_manager.cc:1397] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:15:59.351889 611 log.cc:824] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:59.352768 611 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: No bootstrap required, opened a new log
I20260620 23:15:59.352833 611 ts_tablet_manager.cc:1397] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:15:59.353693 610 raft_consensus.cc:348] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:15:59.353863 610 raft_consensus.cc:374] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:59.353912 610 raft_consensus.cc:729] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3dfef9c251a4f9aae6bf57a7caffce7, State: Initialized, Role: FOLLOWER
I20260620 23:15:59.354033 610 consensus_queue.cc:260] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:15:59.354360 610 ts_tablet_manager.cc:1428] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:15:59.354369 469 heartbeater.cc:503] Master 127.16.79.62:45287 was elected leader, sending a full tablet report...
I20260620 23:15:59.354525 611 raft_consensus.cc:348] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:15:59.354697 611 raft_consensus.cc:374] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:59.354763 611 raft_consensus.cc:729] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dccc888c21a46b0bfd59a29c735a969, State: Initialized, Role: FOLLOWER
I20260620 23:15:59.354863 611 consensus_queue.cc:260] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:15:59.355139 611 ts_tablet_manager.cc:1428] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:15:59.355158 601 heartbeater.cc:503] Master 127.16.79.62:45287 was elected leader, sending a full tablet report...
I20260620 23:15:59.363343 615 raft_consensus.cc:482] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:59.363435 615 raft_consensus.cc:504] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:15:59.363744 615 leader_election.cc:288] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3dfef9c251a4f9aae6bf57a7caffce7 (127.16.79.2:35419), a6844eaef5f244a48f0af6c373274c80 (127.16.79.1:40145)
I20260620 23:15:59.367256 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.363980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:35419 (local address 127.16.79.3:57225)
0620 23:15:59.364047 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:59.364061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:59.364109 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:59.364283 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:59.364285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:59.364287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:59.364353 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.364354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.365222 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.365223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.366301 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.366303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.367015 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.367015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.367108 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.367113 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:15:59.367175 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:15:59.367206 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:15:59.367297 607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.364040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:57225 (local address 127.16.79.2:35419)
0620 23:15:59.364081 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:59.364083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:59.364094 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:15:59.364198 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:59.364199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:59.364209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:59.364274 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:59.364438 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.365128 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.366386 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.366926 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.367001 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.367010 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:59.367184 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 23:15:59.367222 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:15:59.367630 423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d843c57c6a4340b0dc32a372f4d6e5" candidate_uuid: "0dccc888c21a46b0bfd59a29c735a969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" is_pre_election: true
I20260620 23:15:59.367717 616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.363980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:40145 (local address 127.16.79.3:35301)
0620 23:15:59.364484 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:59.364501 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:59.364567 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:59.364779 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:59.364780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:59.364781 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:59.364841 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.364842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.365557 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.365558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.366611 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:59.366613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.367460 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.367461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:59.367536 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.367540 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:15:59.367606 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 23:15:59.367645 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":199,"thread_start_us":111,"threads_started":1}
I20260620 23:15:59.367745 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.364109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:35301 (local address 127.16.79.1:40145)
0620 23:15:59.364177 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:59.364180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:59.364574 (+ 394us) server_negotiation.cc:407] Connection header received
0620 23:15:59.364606 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:59.364607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:59.364619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:59.364676 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:59.364912 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.365471 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.366666 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.367248 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.367315 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.367327 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:59.367629 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 23:15:59.367668 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:15:59.367758 423 raft_consensus.cc:2454] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dccc888c21a46b0bfd59a29c735a969 in term 0.
I20260620 23:15:59.367976 488 leader_election.cc:302] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dccc888c21a46b0bfd59a29c735a969, d3dfef9c251a4f9aae6bf57a7caffce7; no voters:
I20260620 23:15:59.368088 615 raft_consensus.cc:2781] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:15:59.368031 32753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d843c57c6a4340b0dc32a372f4d6e5" candidate_uuid: "0dccc888c21a46b0bfd59a29c735a969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6844eaef5f244a48f0af6c373274c80" is_pre_election: true
I20260620 23:15:59.368148 615 raft_consensus.cc:482] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:15:59.368181 615 raft_consensus.cc:3037] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:59.368197 32753 raft_consensus.cc:2454] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dccc888c21a46b0bfd59a29c735a969 in term 0.
I20260620 23:15:59.368957 615 raft_consensus.cc:504] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:15:59.369099 615 leader_election.cc:288] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [CANDIDATE]: Term 1 election: Requested vote from peers d3dfef9c251a4f9aae6bf57a7caffce7 (127.16.79.2:35419), a6844eaef5f244a48f0af6c373274c80 (127.16.79.1:40145)
I20260620 23:15:59.369201 423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d843c57c6a4340b0dc32a372f4d6e5" candidate_uuid: "0dccc888c21a46b0bfd59a29c735a969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7"
I20260620 23:15:59.369247 32753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d843c57c6a4340b0dc32a372f4d6e5" candidate_uuid: "0dccc888c21a46b0bfd59a29c735a969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6844eaef5f244a48f0af6c373274c80"
I20260620 23:15:59.369282 423 raft_consensus.cc:3037] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:59.369311 32753 raft_consensus.cc:3037] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:59.370234 423 raft_consensus.cc:2454] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dccc888c21a46b0bfd59a29c735a969 in term 1.
I20260620 23:15:59.370510 488 leader_election.cc:302] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dccc888c21a46b0bfd59a29c735a969, d3dfef9c251a4f9aae6bf57a7caffce7; no voters:
I20260620 23:15:59.370661 615 raft_consensus.cc:2781] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:59.370816 615 raft_consensus.cc:686] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 1 LEADER]: Becoming Leader. State: Replica: 0dccc888c21a46b0bfd59a29c735a969, State: Running, Role: LEADER
I20260620 23:15:59.370904 615 consensus_queue.cc:237] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:15:59.370970 32753 raft_consensus.cc:2454] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dccc888c21a46b0bfd59a29c735a969 in term 1.
I20260620 23:15:59.371615 32611 catalog_manager.cc:5495] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dccc888c21a46b0bfd59a29c735a969 (127.16.79.3). New cstate: current_term: 1 leader_uuid: "0dccc888c21a46b0bfd59a29c735a969" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } health_report { overall_health: UNKNOWN } } }
I20260620 23:15:59.381230 608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.378687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39022 (local address 127.16.79.3:39541)
0620 23:15:59.378747 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:59.378749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:59.378757 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:15:59.378806 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:59.378807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:59.378822 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:59.378877 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:59.378962 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.379409 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.380187 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:59.380569 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:59.380630 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:59.380903 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:59.380994 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:15:59.381062 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:59.381087 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 23:15:59.381104 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:15:59.382849 32753 raft_consensus.cc:1261] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Refusing update from remote peer 0dccc888c21a46b0bfd59a29c735a969: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:59.382947 423 raft_consensus.cc:1261] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [term 1 FOLLOWER]: Refusing update from remote peer 0dccc888c21a46b0bfd59a29c735a969: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:59.383208 617 consensus_queue.cc:1035] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:59.383381 615 consensus_queue.cc:1035] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:59.386915 623 mvcc.cc:205] Tried to move back new op lower bound from 7299061184029523968 to 7299061183983849472. Current Snapshot: MvccSnapshot[applied={T|T < 7299061184029523968}]
I20260620 23:15:59.388900 32611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:59.388998 32611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:59.389662 32611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Sending DeleteTablet for 3 replicas of tablet 59d843c57c6a4340b0dc32a372f4d6e5
I20260620 23:15:59.389959 32733 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d843c57c6a4340b0dc32a372f4d6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:59 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:60668
I20260620 23:15:59.389972 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 32649
I20260620 23:15:59.390051 340 tablet_replica.cc:330] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: stopping tablet replica
I20260620 23:15:59.390012 403 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d843c57c6a4340b0dc32a372f4d6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:59 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:38178
I20260620 23:15:59.390093 340 raft_consensus.cc:2229] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:59.390156 340 raft_consensus.cc:2258] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:59.392933 630 tablet_replica.cc:330] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: stopping tablet replica
I20260620 23:15:59.392964 535 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d843c57c6a4340b0dc32a372f4d6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:59 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:39018
I20260620 23:15:59.393035 630 raft_consensus.cc:2229] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:59.393132 630 raft_consensus.cc:2258] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:59.393312 631 tablet_replica.cc:330] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: stopping tablet replica
I20260620 23:15:59.393433 631 raft_consensus.cc:2229] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:15:59.393558 630 ts_tablet_manager.cc:1905] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:59.393599 631 raft_consensus.cc:2258] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:59.395589 630 ts_tablet_manager.cc:1918] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:59.395663 630 log.cc:1196] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/59d843c57c6a4340b0dc32a372f4d6e5
W20260620 23:15:59.395661 32596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:40145 (error 108)
I20260620 23:15:59.395964 630 ts_tablet_manager.cc:1939] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Deleting consensus metadata
I20260620 23:15:59.396158 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 337
I20260620 23:15:59.397042 606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.396928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:40145 (local address 127.0.0.1:60678)
0620 23:15:59.396970 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:59.397011 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 23:15:59.397166 32596 catalog_manager.cc:4584] TS a6844eaef5f244a48f0af6c373274c80 (127.16.79.1:40145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59d843c57c6a4340b0dc32a372f4d6e5: Network error: Client connection negotiation failed: client connection to 127.16.79.1:40145: connect: Connection refused (error 111)
I20260620 23:15:59.401971 631 ts_tablet_manager.cc:1905] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:59.408620 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 472
I20260620 23:15:59.408655 606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.408527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:35419 (local address 127.0.0.1:38188)
0620 23:15:59.408577 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:59.408617 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.2:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 23:15:59.415351 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 32578
I20260620 23:15:59.415385 606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:59.415256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:39541 (local address 127.0.0.1:39038)
0620 23:15:59.415314 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:59.415345 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.3:39541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 23:16:19.420569 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45287
--webserver_interface=127.16.79.62
--webserver_port=42233
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:45287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:19.502640 633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:19.502835 633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:19.502857 633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:19.503975 633 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:16:19.504020 633 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:16:19.504034 633 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:16:19.504046 633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:19.504060 633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:19.504072 633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:19.504086 633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:19.504098 633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:19.505563 633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:45287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=42233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:19.505801 633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:19.506045 633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:16:19.511361 633 server_base.cc:1034] running on GCE node
W20260620 23:16:19.511448 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:19.511477 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:19.511457 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:19.512019 633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:19.512353 633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:19.513561 633 hybrid_clock.cc:648] HybridClock initialized: now 1781997379513549 us; error 29 us; skew 500 ppm
Jun 20 23:16:19 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:19.519145 633 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:16:19.519637 633 webserver.cc:460] Webserver started at http://127.16.79.62:42233/ using document root <none> and password file <none>
I20260620 23:16:19.519868 633 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:19.519938 633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:19.521538 633 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:19.522644 648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:19.523016 633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:19.523155 633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "2465aef6b3ea488cbe792beddcfa9646"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "2b80e05c06bb3bd8def6afc52e15c73e"
server_key_iv: "fc66917b23b642c8aed7d3a41b87d23d"
server_key_version: "encryptionkey@0"
I20260620 23:16:19.523639 633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:19.543498 633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:19.546175 633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:19.546360 633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:19.550588 633 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:45287
I20260620 23:16:19.550624 700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:45287 every 8 connection(s)
I20260620 23:16:19.550997 633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:19.552455 701 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:16:19.553220 701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Bootstrap starting.
I20260620 23:16:19.556587 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 633
I20260620 23:16:19.556949 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:40145
--local_ip_for_outbound_sockets=127.16.79.1
--tserver_master_addrs=127.16.79.62:45287
--webserver_port=42211
--webserver_interface=127.16.79.1
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:16:19.562040 701 log.cc:824] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:19.564126 701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:16:19.564467 701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Bootstrap complete.
I20260620 23:16:19.566694 701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } }
I20260620 23:16:19.566919 701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2465aef6b3ea488cbe792beddcfa9646, State: Initialized, Role: FOLLOWER
I20260620 23:16:19.567070 701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } }
I20260620 23:16:19.567171 701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:19.567221 701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:19.567292 701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:16:19.568229 701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } }
I20260620 23:16:19.568343 701 leader_election.cc:302] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2465aef6b3ea488cbe792beddcfa9646; no voters:
I20260620 23:16:19.568519 701 leader_election.cc:288] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:16:19.568598 705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:16:19.568794 701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:19.568827 705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [term 3 LEADER]: Becoming Leader. State: Replica: 2465aef6b3ea488cbe792beddcfa9646, State: Running, Role: LEADER
I20260620 23:16:19.569067 705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } }
I20260620 23:16:19.569317 705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2465aef6b3ea488cbe792beddcfa9646. Latest consensus state: current_term: 3 leader_uuid: "2465aef6b3ea488cbe792beddcfa9646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } } }
I20260620 23:16:19.569370 705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:19.569800 706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2465aef6b3ea488cbe792beddcfa9646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2465aef6b3ea488cbe792beddcfa9646" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45287 } } }
I20260620 23:16:19.569911 706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:19.570019 712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:19.570883 712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:19.570979 712 catalog_manager.cc:1259] Loaded cluster ID: 8101f7ec69164f12bc7c06c9d0eeef96
I20260620 23:16:19.571010 712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:19.572263 712 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:19.572448 712 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Loaded TSK: 0
I20260620 23:16:19.572490 712 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2465aef6b3ea488cbe792beddcfa9646: Loaded TSK: 1
F20260620 23:16:19.572743 712 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1ea36c19ad google::LogMessage::Fail() at ??:0
@ 0x7f1ea36c488a google::LogMessage::SendToLog() at ??:0
@ 0x7f1ea36c14de google::LogMessage::Flush() at ??:0
@ 0x7f1ea36c2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1ea5e2e104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1ea5e2bd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1eab61b1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1eab61b48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1eab61b594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1eab5e1c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1eab5ebad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1ea558f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1ea5584954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1ea63e86db start_thread at ??:0
@ 0x7f1ea21bf61f clone at ??:0
W20260620 23:16:19.655802 703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:19.656009 703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:19.656046 703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:19.657212 703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:19.657270 703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:19.657321 703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:16:19.658728 703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:40145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=42211
--tserver_master_addrs=127.16.79.62:45287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:19.658986 703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:19.659198 703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:19.664166 745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:19.664137 742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:19.664367 703 server_base.cc:1034] running on GCE node
W20260620 23:16:19.664149 743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:19.664595 703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:19.664875 703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:19.666035 703 hybrid_clock.cc:648] HybridClock initialized: now 1781997379666024 us; error 28 us; skew 500 ppm
Jun 20 23:16:19 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:19.668536 703 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:16:19.669050 703 webserver.cc:460] Webserver started at http://127.16.79.1:42211/ using document root <none> and password file <none>
I20260620 23:16:19.669289 703 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:19.669333 703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:19.670605 703 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:19.671295 752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:19.671447 703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:16:19.671551 703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "a6844eaef5f244a48f0af6c373274c80"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "6c906e0f70bf143d5501f639df7260b8"
server_key_iv: "c17b77a601d7ca43fc0010afa1db4194"
server_key_version: "encryptionkey@0"
I20260620 23:16:19.671959 703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:19.686559 703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:19.688892 703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:19.689044 703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:19.689275 703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:19.689726 759 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 23:16:19.691114 703 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:16:19.691169 703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 23:16:19.691195 703 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:16:19.691826 703 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:16:19.691870 703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 23:16:19.691972 759 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: Bootstrap starting.
I20260620 23:16:19.699939 703 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:40145
I20260620 23:16:19.700116 867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:40145 every 8 connection(s)
I20260620 23:16:19.700423 703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:16:19.702214 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 703
I20260620 23:16:19.702541 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:35419
--local_ip_for_outbound_sockets=127.16.79.2
--tserver_master_addrs=127.16.79.62:45287
--webserver_port=45669
--webserver_interface=127.16.79.2
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:16:19.704083 759 log.cc:824] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:19.706097 759 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:16:19.706435 759 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: Bootstrap complete.
I20260620 23:16:19.707332 759 ts_tablet_manager.cc:1397] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260620 23:16:19.708324 759 raft_consensus.cc:348] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:16:19.708566 759 raft_consensus.cc:729] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6844eaef5f244a48f0af6c373274c80, State: Initialized, Role: FOLLOWER
I20260620 23:16:19.708757 759 consensus_queue.cc:260] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:16:19.709041 759 ts_tablet_manager.cc:1428] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
W20260620 23:16:19.786072 870 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:19.701162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:39035)
0620 23:16:19.701376 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:19.701424 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:19.702011 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:19.785867 (+ 83856us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":107,"threads_started":1}
W20260620 23:16:19.786369 868 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:45287 (0 consecutive failures): Network error: Failed to ping master at 127.16.79.62:45287: Client connection negotiation failed: client connection to 127.16.79.62:45287: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:16:19.786752 870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:19.786632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:60695)
0620 23:16:19.786690 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:19.786735 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:16:19.787124 870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:19.787058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:58537)
0620 23:16:19.787087 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:19.787109 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 23:16:19.787300 868 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:16:19.799778 871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:19.799993 871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:19.800031 871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:19.801236 871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:19.801296 871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:19.801340 871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:16:19.802794 871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:35419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=45669
--tserver_master_addrs=127.16.79.62:45287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:19.803035 871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:19.803252 871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:19.808102 879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:19.808108 881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:19.808112 878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:19.808384 871 server_base.cc:1034] running on GCE node
I20260620 23:16:19.808645 871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:19.808915 871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:19.810050 871 hybrid_clock.cc:648] HybridClock initialized: now 1781997379810032 us; error 38 us; skew 500 ppm
Jun 20 23:16:19 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:19.812495 871 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:16:19.812984 871 webserver.cc:460] Webserver started at http://127.16.79.2:45669/ using document root <none> and password file <none>
I20260620 23:16:19.813221 871 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:19.813288 871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:19.814605 871 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:19.815219 888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:19.815411 871 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:19.815498 871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "d3dfef9c251a4f9aae6bf57a7caffce7"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "7c6a1d4d06384cbb4b89dd481d1ce037"
server_key_iv: "1d29afbf528a5847ad1cfdd9f20db1ff"
server_key_version: "encryptionkey@0"
I20260620 23:16:19.815876 871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:19.826661 871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:19.829190 871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:19.829356 871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:19.829576 871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:19.829871 871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:19.829922 871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:19.829962 871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:19.829988 871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:19.836081 871 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:35419
I20260620 23:16:19.836195 1001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:35419 every 8 connection(s)
I20260620 23:16:19.836479 871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:16:19.837558 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:19.837186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:48445)
0620 23:16:19.837328 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:19.837418 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
W20260620 23:16:19.837882 1002 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:45287 (0 consecutive failures): Network error: Failed to ping master at 127.16.79.62:45287: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
I20260620 23:16:19.838133 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:19.838048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:59623)
0620 23:16:19.838092 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:19.838117 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 23:16:19.838369 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:19.838324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:55899)
0620 23:16:19.838343 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:19.838359 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260620 23:16:19.838486 1002 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:16:19.838711 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 871
I20260620 23:16:19.839156 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:39541
--local_ip_for_outbound_sockets=127.16.79.3
--tserver_master_addrs=127.16.79.62:45287
--webserver_port=33825
--webserver_interface=127.16.79.3
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:19.923708 1005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:19.923902 1005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:19.923928 1005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:19.925036 1005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:19.925084 1005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:19.925216 1005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:16:19.926699 1005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:39541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=33825
--tserver_master_addrs=127.16.79.62:45287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:19.926908 1005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:19.927125 1005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:19.932294 1010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:19.932293 1013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:19.932454 1011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:19.932574 1005 server_base.cc:1034] running on GCE node
I20260620 23:16:19.932901 1005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:19.933176 1005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:19.934335 1005 hybrid_clock.cc:648] HybridClock initialized: now 1781997379934322 us; error 27 us; skew 500 ppm
Jun 20 23:16:19 dist-test-slave-pswb krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:19.937197 1005 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:16:19.937739 1005 webserver.cc:460] Webserver started at http://127.16.79.3:33825/ using document root <none> and password file <none>
I20260620 23:16:19.937994 1005 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:19.938063 1005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:19.939339 1005 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:19.940258 1020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:19.940515 1005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:19.940620 1005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "0dccc888c21a46b0bfd59a29c735a969"
format_stamp: "Formatted at 2026-06-20 23:15:38 on dist-test-slave-pswb"
server_key: "0d7a7db66307bb3442306cd5e4c81acb"
server_key_iv: "97fcf3ef59416019e80c47092d694bdc"
server_key_version: "encryptionkey@0"
I20260620 23:16:19.941015 1005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:19.954159 1005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:19.956226 1005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:19.956386 1005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:19.956594 1005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:19.957049 1027 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 23:16:19.958422 1005 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:16:19.958510 1005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 23:16:19.958551 1005 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:16:19.959179 1005 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:16:19.959247 1005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 23:16:19.959273 1027 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: Bootstrap starting.
I20260620 23:16:19.961264 1027 log.cc:824] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:19.964164 1027 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:16:19.964495 1027 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: Bootstrap complete.
I20260620 23:16:19.964643 1027 ts_tablet_manager.cc:1397] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 23:16:19.965801 1005 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:39541
I20260620 23:16:19.965902 1135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:39541 every 8 connection(s)
I20260620 23:16:19.966158 1005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:16:19.966444 1027 raft_consensus.cc:348] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:16:19.966825 1027 raft_consensus.cc:729] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dccc888c21a46b0bfd59a29c735a969, State: Initialized, Role: FOLLOWER
I20260620 23:16:19.966949 1027 consensus_queue.cc:260] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:16:19.967200 1027 ts_tablet_manager.cc:1428] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:16:19.967252 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:19.966833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:39247)
0620 23:16:19.967000 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:19.967097 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
W20260620 23:16:19.967404 1136 heartbeater.cc:644] Failed to heartbeat to 127.16.79.62:45287 (0 consecutive failures): Network error: Failed to ping master at 127.16.79.62:45287: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
I20260620 23:16:19.967725 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:19.967573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:40317)
0620 23:16:19.967672 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:19.967702 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 23:16:19.968014 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:19.967957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:36069)
0620 23:16:19.967979 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:19.968000 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 23:16:19.968125 1136 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:16:19.976083 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 1005
I20260620 23:16:20.787861 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:20.787610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:48397)
0620 23:16:20.787762 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:20.787801 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260620 23:16:20.839061 1142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:20.838823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:33893)
0620 23:16:20.838980 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:20.839016 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260620 23:16:20.968847 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:20.968478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:48807)
0620 23:16:20.968635 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:20.968678 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260620 23:16:21.244910 1144 raft_consensus.cc:482] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:16:21.245048 1144 raft_consensus.cc:504] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:16:21.245393 1144 leader_election.cc:288] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0dccc888c21a46b0bfd59a29c735a969 (127.16.79.3:39541), d3dfef9c251a4f9aae6bf57a7caffce7 (127.16.79.2:35419)
Jun 20 23:16:21 dist-test-slave-pswb krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.3@KRBTEST.COM
Jun 20 23:16:21 dist-test-slave-pswb krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.2@KRBTEST.COM
I20260620 23:16:21.256196 1147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:21.245766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:47953 (local address 127.16.79.2:35419)
0620 23:16:21.245936 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:21.245953 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:21.245987 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:16:21.246680 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:21.246682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:21.246720 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:21.246882 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:21.247295 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:21.248225 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:21.249227 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:21.249760 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:21.249961 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:21.252455 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:21.252459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:21.252459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:21.252468 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:21.254508 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:21.255043 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:21.255046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:21.255046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:21.255138 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:21.255360 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:21.255361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:21.255361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:21.255599 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:21.255687 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:21.255953 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:16:21.256079 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260620 23:16:21.256187 1146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:21.245664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:35419 (local address 127.16.79.1:47953)
0620 23:16:21.245832 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:21.245845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:21.245893 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:21.246888 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:21.246889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:21.247105 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:21.247174 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:21.247176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:21.248330 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:21.248331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:21.249129 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:21.249131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:21.249929 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:21.249930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:21.249997 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:21.250609 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:21.250614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:21.252336 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:21.254648 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:21.254649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:21.254650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:21.254961 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:21.255218 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:21.255219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:21.255219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:21.255287 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:21.255684 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:21.255685 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:21.255799 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 23:16:21.255901 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:16:21.256092 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260620 23:16:21.256223 1145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:21.245672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:32951 (local address 127.16.79.3:39541)
0620 23:16:21.245903 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:21.245920 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:21.245955 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:16:21.246129 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:21.246132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:21.246194 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:21.246388 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:21.247139 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:21.248057 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:21.249232 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:21.249632 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:21.249764 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:21.252092 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:21.252096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:21.252096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:21.252106 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:21.254359 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:21.254956 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:21.254958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:21.254958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:21.255080 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:21.255346 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:21.255347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:21.255347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:21.255587 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:21.255681 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:21.255954 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 23:16:21.256086 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260620 23:16:21.256600 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:21.245533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:39541 (local address 127.16.79.1:32951)
0620 23:16:21.245623 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:21.245635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:21.245699 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:21.246463 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:21.246466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:21.246788 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:21.246982 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:21.246989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:21.248182 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:21.248183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:21.249138 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:21.249140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:21.249730 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:21.249731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:21.249858 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:21.250442 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:21.250452 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:21.251965 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:21.254464 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:21.254470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:21.254472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:21.254794 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:21.255161 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:21.255162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:21.255162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:21.255277 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:21.255662 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:21.255668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:21.255796 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 23:16:21.255900 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:16:21.256171 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":155}
W20260620 23:16:21.257249 753 leader_election.cc:341] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3dfef9c251a4f9aae6bf57a7caffce7 (127.16.79.2:35419): Not found: Tablet not found: 59d843c57c6a4340b0dc32a372f4d6e5
I20260620 23:16:21.257341 1090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d843c57c6a4340b0dc32a372f4d6e5" candidate_uuid: "a6844eaef5f244a48f0af6c373274c80" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0dccc888c21a46b0bfd59a29c735a969" is_pre_election: true
I20260620 23:16:21.257779 1090 raft_consensus.cc:2454] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6844eaef5f244a48f0af6c373274c80 in term 1.
I20260620 23:16:21.258162 755 leader_election.cc:302] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0dccc888c21a46b0bfd59a29c735a969, a6844eaef5f244a48f0af6c373274c80; no voters: d3dfef9c251a4f9aae6bf57a7caffce7
I20260620 23:16:21.258347 1144 raft_consensus.cc:2781] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:16:21.258427 1144 raft_consensus.cc:482] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:16:21.258451 1144 raft_consensus.cc:3037] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:16:21.259624 1144 raft_consensus.cc:504] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:16:21.259778 1144 leader_election.cc:288] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [CANDIDATE]: Term 2 election: Requested vote from peers 0dccc888c21a46b0bfd59a29c735a969 (127.16.79.3:39541), d3dfef9c251a4f9aae6bf57a7caffce7 (127.16.79.2:35419)
I20260620 23:16:21.259956 1090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d843c57c6a4340b0dc32a372f4d6e5" candidate_uuid: "a6844eaef5f244a48f0af6c373274c80" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0dccc888c21a46b0bfd59a29c735a969"
I20260620 23:16:21.260078 1090 raft_consensus.cc:3037] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 1 FOLLOWER]: Advancing to term 2
W20260620 23:16:21.260125 753 leader_election.cc:341] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer d3dfef9c251a4f9aae6bf57a7caffce7 (127.16.79.2:35419): Not found: Tablet not found: 59d843c57c6a4340b0dc32a372f4d6e5
I20260620 23:16:21.261313 1090 raft_consensus.cc:2454] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6844eaef5f244a48f0af6c373274c80 in term 2.
I20260620 23:16:21.261696 755 leader_election.cc:302] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0dccc888c21a46b0bfd59a29c735a969, a6844eaef5f244a48f0af6c373274c80; no voters: d3dfef9c251a4f9aae6bf57a7caffce7
I20260620 23:16:21.262061 1144 raft_consensus.cc:2781] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:16:21.262181 1144 raft_consensus.cc:686] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [term 2 LEADER]: Becoming Leader. State: Replica: a6844eaef5f244a48f0af6c373274c80, State: Running, Role: LEADER
I20260620 23:16:21.262290 1144 consensus_queue.cc:237] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:16:21.262699 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:21.262499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:34933)
0620 23:16:21.262555 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:21.262604 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 23:16:21.657487 1090 raft_consensus.cc:1261] T 59d843c57c6a4340b0dc32a372f4d6e5 P 0dccc888c21a46b0bfd59a29c735a969 [term 2 FOLLOWER]: Refusing update from remote peer a6844eaef5f244a48f0af6c373274c80: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 23:16:21.657888 1144 consensus_queue.cc:1035] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 23:16:21.659215 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:21.658956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:58103)
0620 23:16:21.659094 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:21.659171 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":104,"threads_started":1}
W20260620 23:16:21.660007 753 consensus_peers.cc:487] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80 -> Peer d3dfef9c251a4f9aae6bf57a7caffce7 (127.16.79.2:35419): Couldn't send request to peer d3dfef9c251a4f9aae6bf57a7caffce7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59d843c57c6a4340b0dc32a372f4d6e5. This is attempt 1: this message will repeat every 5th retry.
I20260620 23:16:21.660100 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:21.659993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:51213)
0620 23:16:21.660032 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:21.660065 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 23:16:21.660429 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:21.660346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:48687)
0620 23:16:21.660378 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:21.660401 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 23:16:21.839902 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:21.839663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:50017)
0620 23:16:21.839823 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:21.839865 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260620 23:16:22.249339 1158 ts_tablet_manager.cc:927] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Initiating tablet copy from peer a6844eaef5f244a48f0af6c373274c80 (127.16.79.1:40145)
I20260620 23:16:22.249641 1158 tablet_copy_client.cc:316] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.79.1:40145
Jun 20 23:16:22 dist-test-slave-pswb krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.1@KRBTEST.COM
I20260620 23:16:22.257997 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:22.249889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:40145 (local address 127.16.79.2:48775)
0620 23:16:22.249968 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:22.249987 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:22.250071 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:22.250305 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:22.250310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:22.250587 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:22.250656 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:22.250658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:22.251570 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:22.251571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:22.252166 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:22.252168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:22.252641 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:22.252642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:22.252755 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:22.253345 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:22.253350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:22.254732 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:22.256408 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:22.256415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:22.256417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:22.257056 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:22.257352 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:22.257353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:22.257354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:22.257432 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:22.257704 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:22.257710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:22.257752 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:16:22.257819 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:16:22.257906 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 23:16:22.258019 1160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:22.250048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:48775 (local address 127.16.79.1:40145)
0620 23:16:22.250159 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:22.250164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:22.250177 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:16:22.250205 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:22.250207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:22.250218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:22.250280 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:22.250764 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:22.251454 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:22.252265 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:22.252566 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:22.252625 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:22.254812 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:22.254818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:22.254819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:22.254825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:22.256297 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:22.257185 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:22.257188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:22.257188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:22.257249 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:22.257499 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:22.257499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:22.257500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:22.257637 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:22.257703 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:22.257882 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 23:16:22.257948 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":64,"threads_started":1}
I20260620 23:16:22.258448 842 tablet_copy_service.cc:140] P a6844eaef5f244a48f0af6c373274c80: Received BeginTabletCopySession request for tablet 59d843c57c6a4340b0dc32a372f4d6e5 from peer d3dfef9c251a4f9aae6bf57a7caffce7 ({username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:48775)
I20260620 23:16:22.258507 842 tablet_copy_service.cc:161] P a6844eaef5f244a48f0af6c373274c80: Beginning new tablet copy session on tablet 59d843c57c6a4340b0dc32a372f4d6e5 from peer d3dfef9c251a4f9aae6bf57a7caffce7 at {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:48775: session id = d3dfef9c251a4f9aae6bf57a7caffce7-59d843c57c6a4340b0dc32a372f4d6e5
I20260620 23:16:22.258966 842 tablet_copy_source_session.cc:205] T 59d843c57c6a4340b0dc32a372f4d6e5 P a6844eaef5f244a48f0af6c373274c80: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:16:22.260087 1158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d843c57c6a4340b0dc32a372f4d6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:22.263233 1158 tablet_copy_client.cc:791] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: tablet copy: Starting download of 0 data blocks...
I20260620 23:16:22.263396 1158 tablet_copy_client.cc:655] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: tablet copy: Starting download of 1 WAL segments...
I20260620 23:16:22.264475 1158 tablet_copy_client.cc:531] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:16:22.265556 1158 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Bootstrap starting.
I20260620 23:16:22.267144 1158 log.cc:824] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:22.270030 1158 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:16:22.270288 1158 tablet_bootstrap.cc:492] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Bootstrap complete.
I20260620 23:16:22.270442 1158 ts_tablet_manager.cc:1397] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260620 23:16:22.271420 1158 raft_consensus.cc:348] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:16:22.271595 1158 raft_consensus.cc:729] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3dfef9c251a4f9aae6bf57a7caffce7, State: Initialized, Role: FOLLOWER
I20260620 23:16:22.271728 1158 consensus_queue.cc:260] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dccc888c21a46b0bfd59a29c735a969" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 39541 } } peers { permanent_uuid: "d3dfef9c251a4f9aae6bf57a7caffce7" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35419 } } peers { permanent_uuid: "a6844eaef5f244a48f0af6c373274c80" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40145 } }
I20260620 23:16:22.271996 1158 ts_tablet_manager.cc:1428] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260620 23:16:22.272287 842 tablet_copy_service.cc:342] P a6844eaef5f244a48f0af6c373274c80: Request end of tablet copy session d3dfef9c251a4f9aae6bf57a7caffce7-59d843c57c6a4340b0dc32a372f4d6e5 received from {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:48775
I20260620 23:16:22.272351 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:22.272210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:53069)
0620 23:16:22.272273 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:22.272308 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:16:22.272363 842 tablet_copy_service.cc:433] P a6844eaef5f244a48f0af6c373274c80: ending tablet copy session d3dfef9c251a4f9aae6bf57a7caffce7-59d843c57c6a4340b0dc32a372f4d6e5 on tablet 59d843c57c6a4340b0dc32a372f4d6e5 with peer d3dfef9c251a4f9aae6bf57a7caffce7
I20260620 23:16:22.661280 1163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:22.660948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:48845)
0620 23:16:22.661188 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:22.661224 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":157,"threads_started":1}
I20260620 23:16:22.661345 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:22.661104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:33131)
0620 23:16:22.661247 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:22.661301 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260620 23:16:22.835132 1163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:22.835018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:52697)
0620 23:16:22.835065 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:22.835095 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 23:16:22.835166 956 raft_consensus.cc:1203] T 59d843c57c6a4340b0dc32a372f4d6e5 P d3dfef9c251a4f9aae6bf57a7caffce7 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 23:16:22.835759 1167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:22.835612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:48551)
0620 23:16:22.835691 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:22.835720 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65,"thread_start_us":47,"threads_started":1}
I20260620 23:16:23.662098 1168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:23.661821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:51419)
0620 23:16:23.662002 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:23.662049 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
I20260620 23:16:23.835887 1169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:23.835621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:43807)
0620 23:16:23.835784 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:23.835828 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260620 23:16:23.836326 1170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:23.836097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:51215)
0620 23:16:23.836242 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:23.836281 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260620 23:16:24.663089 1172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:24.662758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:39643)
0620 23:16:24.662994 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:24.663037 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":128,"threads_started":1}
I20260620 23:16:24.836970 1173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:24.836673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:36433)
0620 23:16:24.836875 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:24.836924 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":138,"threads_started":1}
I20260620 23:16:24.837112 1174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:24.836848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:54087)
0620 23:16:24.836961 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:24.836994 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260620 23:16:24.837446 1173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:24.837301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:49591)
0620 23:16:24.837402 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:24.837425 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:16:25.664106 1175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:25.663799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:51587)
0620 23:16:25.663990 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:25.664039 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260620 23:16:25.838032 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:25.837788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:34045)
0620 23:16:25.837905 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:25.837944 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260620 23:16:25.838286 1177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:25.838014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:46277)
0620 23:16:25.838159 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:25.838207 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260620 23:16:26.665288 1179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:26.664966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:44189)
0620 23:16:26.665183 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:26.665232 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":125,"threads_started":1}
I20260620 23:16:26.838786 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:26.838577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:56403)
0620 23:16:26.838704 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:26.838754 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260620 23:16:26.838896 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:26.838715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:35149)
0620 23:16:26.838817 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:26.838857 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
I20260620 23:16:27.666344 1182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:27.666013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:43057)
0620 23:16:27.666218 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:27.666271 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":140,"threads_started":1}
I20260620 23:16:27.839715 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:27.839371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:52503)
0620 23:16:27.839621 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:27.839661 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":150,"threads_started":1}
I20260620 23:16:27.839946 1184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:27.839658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:54541)
0620 23:16:27.839820 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:27.839888 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260620 23:16:28.667212 1186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:28.666953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:40445)
0620 23:16:28.667111 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:28.667159 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260620 23:16:28.840751 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:28.840431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:51511)
0620 23:16:28.840597 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:28.840647 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260620 23:16:28.840791 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:28.840424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:45655)
0620 23:16:28.840632 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:28.840669 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
I20260620 23:16:29.668306 1189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:29.667967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:36155)
0620 23:16:29.668182 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:29.668234 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":117,"threads_started":1}
I20260620 23:16:29.841634 1190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:29.841350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:57719)
0620 23:16:29.841541 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:29.841581 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":125,"threads_started":1}
I20260620 23:16:29.841794 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:29.841537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:58289)
0620 23:16:29.841694 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:29.841739 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260620 23:16:30.669205 1192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:30.668983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:50593)
0620 23:16:30.669123 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:30.669159 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260620 23:16:30.842543 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:30.842277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:50663)
0620 23:16:30.842457 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:30.842498 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
I20260620 23:16:30.842622 1193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:30.842240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:49877)
0620 23:16:30.842524 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:30.842568 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":169,"threads_started":1}
I20260620 23:16:31.670014 1195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:31.669794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:37603)
0620 23:16:31.669928 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:31.669967 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260620 23:16:31.843358 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:31.843117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:34515)
0620 23:16:31.843261 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:31.843296 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260620 23:16:31.843581 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:31.843196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:50667)
0620 23:16:31.843379 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:31.843432 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":128,"threads_started":1}
I20260620 23:16:32.670828 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:32.670589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:60925)
0620 23:16:32.670740 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:32.670780 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":61,"threads_started":1}
I20260620 23:16:32.844352 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:32.844030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:37897)
0620 23:16:32.844249 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:32.844298 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
I20260620 23:16:32.844354 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:32.844122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:54697)
0620 23:16:32.844251 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:32.844298 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260620 23:16:33.671815 1202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:33.671502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:55977)
0620 23:16:33.671728 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:33.671768 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":92,"threads_started":1}
I20260620 23:16:33.845134 1203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:33.844938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:40325)
0620 23:16:33.845059 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:33.845098 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260620 23:16:33.845124 1204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:33.844929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:44763)
0620 23:16:33.845053 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:33.845077 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260620 23:16:34.672679 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:34.672301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.3:44941)
0620 23:16:34.672544 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:34.672611 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260620 23:16:34.845914 1206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:34.845710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.2:36591)
0620 23:16:34.845843 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:34.845876 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260620 23:16:34.845916 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:34.845722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45287 (local address 127.16.79.1:38263)
0620 23:16:34.845845 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:34.845877 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.62:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260620 23:16:34.980119 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 703
I20260620 23:16:34.985409 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 871
I20260620 23:16:34.989859 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 1005
2026-06-20T23:16:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58223 ms)
[----------] 30 tests from SecurityITest (231190 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:35 dist-test-slave-pswb krb5kdc[1210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:35 dist-test-slave-pswb krb5kdc[1210](info): set up 2 sockets
Jun 20 23:16:35 dist-test-slave-pswb krb5kdc[1210](info): commencing operation
krb5kdc: starting...
W20260620 23:16:36.052558 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:36 dist-test-slave-pswb krb5kdc[1210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:16:36.176627 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:35339
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:35339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:36.259505 1223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:36.259698 1223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:36.259719 1223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:36.260764 1223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:36.260797 1223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:36.260810 1223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:36.260824 1223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:36.260836 1223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:36.262174 1223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:35339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:35339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:36.262589 1223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:36.262908 1223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:36.269093 1229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:36.269107 1228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:36.269107 1231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:36.269289 1223 server_base.cc:1034] running on GCE node
I20260620 23:16:36.269682 1223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:36.269999 1223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:36.271170 1223 hybrid_clock.cc:648] HybridClock initialized: now 1781997396271158 us; error 51 us; skew 500 ppm
Jun 20 23:16:36 dist-test-slave-pswb krb5kdc[1210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:36.273797 1223 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:16:36.274288 1223 webserver.cc:460] Webserver started at http://127.16.79.62:32839/ using document root <none> and password file <none>
I20260620 23:16:36.274521 1223 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:36.274590 1223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:36.274734 1223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:36.275744 1223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "17a9406aa6ce476ba91d2ccac8785154"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-pswb"
server_key: "f4e58c5089cf340cac532e49780b7d5d"
server_key_iv: "fbccde10997e755f1f71aa6c9d2f9acd"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.276089 1223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "17a9406aa6ce476ba91d2ccac8785154"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-pswb"
server_key: "f4e58c5089cf340cac532e49780b7d5d"
server_key_iv: "fbccde10997e755f1f71aa6c9d2f9acd"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.277637 1223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:16:36.278999 1238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:36.279323 1223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 23:16:36.279464 1223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "17a9406aa6ce476ba91d2ccac8785154"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-pswb"
server_key: "f4e58c5089cf340cac532e49780b7d5d"
server_key_iv: "fbccde10997e755f1f71aa6c9d2f9acd"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.279556 1223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:36.297767 1223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:36.300030 1223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:36.300199 1223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:36.305787 1223 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:35339
I20260620 23:16:36.305830 1290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:35339 every 8 connection(s)
I20260620 23:16:36.306272 1223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:36.307219 1291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:36.310249 1291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154: Bootstrap starting.
I20260620 23:16:36.311043 1291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:36.311404 1291 log.cc:824] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:36.312263 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 1223
I20260620 23:16:36.312402 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:36.312533 1291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154: No bootstrap required, opened a new log
I20260620 23:16:36.312597 16700 external_mini_cluster.cc:1428] Setting key decfa67aa3e51e268679046352215777
I20260620 23:16:36.314566 1291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a9406aa6ce476ba91d2ccac8785154" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35339 } }
I20260620 23:16:36.314738 1291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:36.314802 1291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a9406aa6ce476ba91d2ccac8785154, State: Initialized, Role: FOLLOWER
I20260620 23:16:36.314962 1291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a9406aa6ce476ba91d2ccac8785154" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35339 } }
I20260620 23:16:36.315029 1291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:36.315050 1291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:36.315104 1291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:36.316007 1291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a9406aa6ce476ba91d2ccac8785154" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35339 } }
I20260620 23:16:36.316128 1291 leader_election.cc:302] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17a9406aa6ce476ba91d2ccac8785154; no voters:
I20260620 23:16:36.316329 1291 leader_election.cc:288] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:36.316349 1296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:36.316457 1296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [term 1 LEADER]: Becoming Leader. State: Replica: 17a9406aa6ce476ba91d2ccac8785154, State: Running, Role: LEADER
I20260620 23:16:36.316536 1296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a9406aa6ce476ba91d2ccac8785154" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35339 } }
I20260620 23:16:36.316589 1291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:36.316869 1296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17a9406aa6ce476ba91d2ccac8785154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a9406aa6ce476ba91d2ccac8785154" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35339 } } }
I20260620 23:16:36.316942 1296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:36.317071 1296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17a9406aa6ce476ba91d2ccac8785154. Latest consensus state: current_term: 1 leader_uuid: "17a9406aa6ce476ba91d2ccac8785154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a9406aa6ce476ba91d2ccac8785154" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 35339 } } }
I20260620 23:16:36.317186 1300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:36.317222 1296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:36.317732 1300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:16:36 dist-test-slave-pswb krb5kdc[1210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:16:36.321053 1300 catalog_manager.cc:1347] Generated new cluster ID: c4c38527348c4d23af53a7eb69386f7e
I20260620 23:16:36.321106 1300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:36.324568 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.313149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.16.79.62:35339)
0620 23:16:36.313346 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.313353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.313377 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:16:36.314598 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.314607 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.314673 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.314821 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.315568 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.316774 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.317827 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.318512 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.318652 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.321341 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.321345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.321345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.321357 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.323044 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.323420 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.323423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.323423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.323534 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.323745 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.323746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.323747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.323984 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.324164 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.324230 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.324393 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260620 23:16:36.335126 1300 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:36.335801 1300 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:36.346863 1300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154: Generated new TSK 0
I20260620 23:16:36.347141 1300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:16:36.375339 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:35339
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:36.464191 1317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:36.464452 1317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:36.464497 1317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:36.465652 1317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:36.465721 1317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:36.465766 1317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:16:36.467188 1317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:35339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:36.467453 1317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:36.467677 1317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:36.472628 1322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:36.472631 1325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:36.472657 1323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:36.472759 1317 server_base.cc:1034] running on GCE node
I20260620 23:16:36.473060 1317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:36.473242 1317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:36.474372 1317 hybrid_clock.cc:648] HybridClock initialized: now 1781997396474366 us; error 29 us; skew 500 ppm
Jun 20 23:16:36 dist-test-slave-pswb krb5kdc[1210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:36.476732 1317 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:16:36.477128 1317 webserver.cc:460] Webserver started at http://127.16.79.1:35441/ using document root <none> and password file <none>
I20260620 23:16:36.477340 1317 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:36.477406 1317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:36.477551 1317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:36.478585 1317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "76e90fc418484dc3a81fcb43469ec213"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-pswb"
server_key: "6207d89dc9990777eafb1c68dad3f825"
server_key_iv: "a2f9fd377dd65e067330ba4a7c82c595"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.478942 1317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "76e90fc418484dc3a81fcb43469ec213"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-pswb"
server_key: "6207d89dc9990777eafb1c68dad3f825"
server_key_iv: "a2f9fd377dd65e067330ba4a7c82c595"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.480286 1317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:16:36.481112 1332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:36.481294 1317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:16:36.481402 1317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "76e90fc418484dc3a81fcb43469ec213"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-pswb"
server_key: "6207d89dc9990777eafb1c68dad3f825"
server_key_iv: "a2f9fd377dd65e067330ba4a7c82c595"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.481484 1317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:36.498792 1317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:36.501189 1317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:36.501354 1317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:36.501579 1317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:36.501902 1317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:36.501958 1317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:36.501992 1317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:36.502023 1317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:36.508978 1317 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:34447
I20260620 23:16:36.509074 1445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:34447 every 8 connection(s)
I20260620 23:16:36.509367 1317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:16:36.510028 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 1317
I20260620 23:16:36.510128 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:16:36.510351 16700 external_mini_cluster.cc:1428] Setting key 482df2b7e3b32d5dc0d13642f0f9d20f
Jun 20 23:16:36 dist-test-slave-pswb krb5kdc[1210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
I20260620 23:16:36.520995 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.510066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:38789 (local address 127.16.79.62:35339)
0620 23:16:36.510120 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.510123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.510415 (+ 292us) server_negotiation.cc:407] Connection header received
0620 23:16:36.511670 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.511671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.511682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.511733 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.512574 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.513418 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.514409 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.514766 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.514844 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.517492 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.517493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.517493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.517500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.519344 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.519834 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.519836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.519836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.519901 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.520252 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.520253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.520253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.520362 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.520419 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.520760 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.520862 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:16:36.521253 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.510065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:35339 (local address 127.16.79.1:38789)
0620 23:16:36.510247 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.510298 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.511567 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.511803 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.511806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.512183 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:36.512475 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.512481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.513527 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.513528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.514310 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.514312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.515016 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.515017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.515116 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.515635 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:36.515649 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:36.517324 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:36.519505 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.519512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.519513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.519732 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.520077 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.520078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.520078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.520182 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.520463 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:36.520469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:36.520583 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 23:16:36.520680 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:16:36.521010 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
I20260620 23:16:36.521868 1446 heartbeater.cc:347] Connected to a master server at 127.16.79.62:35339
I20260620 23:16:36.521962 1446 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:36.522156 1446 heartbeater.cc:511] Master 127.16.79.62:35339 requested a full tablet report, sending...
I20260620 23:16:36.523604 1255 ts_manager.cc:194] Registered new tserver with Master: 76e90fc418484dc3a81fcb43469ec213 (127.16.79.1:34447)
I20260620 23:16:36.524214 1255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:38789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:16:36.539105 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:35339
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:36.621547 1451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:36.621748 1451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:36.621771 1451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:36.622895 1451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:36.622939 1451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:36.622965 1451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:16:36.624393 1451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:35339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:36.624637 1451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:36.624920 1451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:36.629765 1456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:36.629885 1457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:36.629890 1459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:36.630158 1451 server_base.cc:1034] running on GCE node
I20260620 23:16:36.630332 1451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:36.630511 1451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:36.631660 1451 hybrid_clock.cc:648] HybridClock initialized: now 1781997396631647 us; error 27 us; skew 500 ppm
Jun 20 23:16:36 dist-test-slave-pswb krb5kdc[1210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:36.634872 1451 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:16:36.635347 1451 webserver.cc:460] Webserver started at http://127.16.79.2:33503/ using document root <none> and password file <none>
I20260620 23:16:36.635571 1451 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:36.635646 1451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:36.635795 1451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:36.636858 1451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "c5652b9c42cf4fdaae407db6b048b420"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-pswb"
server_key: "4794571312af4ff0107e52d5a261973f"
server_key_iv: "8259be7712f73b600d2ab3caf9147c0c"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.637200 1451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "c5652b9c42cf4fdaae407db6b048b420"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-pswb"
server_key: "4794571312af4ff0107e52d5a261973f"
server_key_iv: "8259be7712f73b600d2ab3caf9147c0c"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.638495 1451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:16:36.639256 1466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:36.639451 1451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:16:36.639545 1451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "c5652b9c42cf4fdaae407db6b048b420"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-pswb"
server_key: "4794571312af4ff0107e52d5a261973f"
server_key_iv: "8259be7712f73b600d2ab3caf9147c0c"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.639632 1451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:36.658253 1451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:36.660418 1451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:36.660588 1451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:36.660846 1451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:36.661161 1451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:36.661247 1451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:36.661288 1451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:36.661321 1451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:36.668069 1451 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:36043
I20260620 23:16:36.668102 1580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:36043 every 8 connection(s)
I20260620 23:16:36.668440 1451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:16:36 dist-test-slave-pswb krb5kdc[1210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:16:36.674748 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 1451
I20260620 23:16:36.674857 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:16:36.675055 16700 external_mini_cluster.cc:1428] Setting key 6dbe7d39388565da3a5478ff884bbd15
I20260620 23:16:36.678503 1583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.669212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:35339 (local address 127.16.79.2:49485)
0620 23:16:36.669380 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.669421 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.669910 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.670156 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.670160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.670713 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:36.670899 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.670903 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.671606 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.671607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.672409 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.672411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.672860 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.672862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.673038 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.673561 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:36.673586 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:36.675082 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:36.676835 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.676841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.676844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.677115 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.677387 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.677388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.677388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.677539 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.677781 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:36.677787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:36.677974 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 23:16:36.678106 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 23:16:36.678303 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260620 23:16:36.678570 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.669231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:49485 (local address 127.16.79.62:35339)
0620 23:16:36.669299 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.669301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.669498 (+ 197us) server_negotiation.cc:407] Connection header received
0620 23:16:36.670022 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.670023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.670036 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.670092 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.671005 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.671513 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.672515 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.672773 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.672851 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.675189 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.675191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.675192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.675199 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.676664 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.677211 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.677213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.677214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.677282 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.677613 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.677613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.677614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.677713 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.677765 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.678283 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.678469 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:16:36.679194 1581 heartbeater.cc:347] Connected to a master server at 127.16.79.62:35339
I20260620 23:16:36.679311 1581 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:36.679500 1581 heartbeater.cc:511] Master 127.16.79.62:35339 requested a full tablet report, sending...
I20260620 23:16:36.679970 1255 ts_manager.cc:194] Registered new tserver with Master: c5652b9c42cf4fdaae407db6b048b420 (127.16.79.2:36043)
I20260620 23:16:36.680356 1255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:49485
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:16:36.704985 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:35339
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:36.786899 1586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:36.787086 1586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:36.787109 1586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:36.788213 1586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:36.788254 1586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:36.788280 1586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:16:36.789742 1586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:35339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:36.790225 1586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:36.790592 1586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:36.796269 1592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:36.796322 1591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:36.796332 1594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:36.796332 1586 server_base.cc:1034] running on GCE node
I20260620 23:16:36.796828 1586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:36.797024 1586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:36.798177 1586 hybrid_clock.cc:648] HybridClock initialized: now 1781997396798151 us; error 61 us; skew 500 ppm
Jun 20 23:16:36 dist-test-slave-pswb krb5kdc[1210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:36.801237 1586 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:16:36.801687 1586 webserver.cc:460] Webserver started at http://127.16.79.3:42985/ using document root <none> and password file <none>
I20260620 23:16:36.801923 1586 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:36.801971 1586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:36.802098 1586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:36.803134 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "34e15b33a3d5414d8d6cecb43befb62d"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-pswb"
server_key: "9b6699a0816f40d9961fd310f2ccb985"
server_key_iv: "c115cdc08f9bdba82989027e4a96f4d5"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.803484 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "34e15b33a3d5414d8d6cecb43befb62d"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-pswb"
server_key: "9b6699a0816f40d9961fd310f2ccb985"
server_key_iv: "c115cdc08f9bdba82989027e4a96f4d5"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.804840 1586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:16:36.805771 1601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:36.806157 1586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:36.806238 1586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "34e15b33a3d5414d8d6cecb43befb62d"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-pswb"
server_key: "9b6699a0816f40d9961fd310f2ccb985"
server_key_iv: "c115cdc08f9bdba82989027e4a96f4d5"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.806334 1586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:36.833031 1586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:36.835379 1586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:36.835563 1586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:36.835918 1586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:36.836251 1586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:36.836311 1586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:36.836355 1586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:36.836383 1586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:36.842404 1586 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:44089
I20260620 23:16:36.842461 1714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:44089 every 8 connection(s)
I20260620 23:16:36.842751 1586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:16:36 dist-test-slave-pswb krb5kdc[1210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:16:36.850085 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 1586
I20260620 23:16:36.850253 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:16:36.850514 16700 external_mini_cluster.cc:1428] Setting key b14cb38aab456af3bc35f93ad8e693af
I20260620 23:16:36.853602 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.843415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:38911 (local address 127.16.79.62:35339)
0620 23:16:36.843488 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.843491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.843708 (+ 217us) server_negotiation.cc:407] Connection header received
0620 23:16:36.844192 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.844193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.844203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.844255 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.845207 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.845707 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.846682 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.846978 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.847033 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.849598 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.849601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.849601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.849611 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.851868 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.852402 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.852403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.852403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.852463 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.852881 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.852882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.852882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.852996 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.853079 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.853470 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.853538 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:16:36.854033 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.843396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:35339 (local address 127.16.79.3:38911)
0620 23:16:36.843575 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.843619 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.844076 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.844311 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.844314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.844913 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:36.845096 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.845108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.845817 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.845818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.846600 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.846602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.847076 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.847077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.847219 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.847712 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:36.847720 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:36.849303 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:36.851984 (+ 2681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.851993 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.851996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.852300 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.852584 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.852586 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.852586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.852779 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.853127 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:36.853134 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:36.853303 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 23:16:36.853425 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 23:16:36.853594 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260620 23:16:36.854962 1715 heartbeater.cc:347] Connected to a master server at 127.16.79.62:35339
I20260620 23:16:36.855049 1715 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:36.855245 1715 heartbeater.cc:511] Master 127.16.79.62:35339 requested a full tablet report, sending...
I20260620 23:16:36.855602 1255 ts_manager.cc:194] Registered new tserver with Master: 34e15b33a3d5414d8d6cecb43befb62d (127.16.79.3:44089)
I20260620 23:16:36.855988 1255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:38911
I20260620 23:16:36.863124 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:16:36.870648 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.864513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.16.79.62:35339)
0620 23:16:36.864551 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.864552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.864655 (+ 103us) server_negotiation.cc:407] Connection header received
0620 23:16:36.864772 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.864775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.864788 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.864851 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.865393 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.865872 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.866590 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.866762 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.866816 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.867886 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.867887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.867887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.867893 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.869314 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.869607 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.869607 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.869608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.869654 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.869968 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.869970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.869970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.870208 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.870333 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.870486 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.870576 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:16:36.872293 1255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:16:36.873917 1255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:16:36.880563 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.876404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:44089 (local address 127.0.0.1:44926)
0620 23:16:36.876713 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.876753 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.876831 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.877137 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.877142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.877144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:36.877251 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.877254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.878550 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.878551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.879600 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.879601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.880249 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.880250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.880330 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.880343 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:16:36.880408 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 23:16:36.880465 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":8,"thread_start_us":42,"threads_started":1}
I20260620 23:16:36.880834 1731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.876815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.16.79.1:34447)
0620 23:16:36.877109 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.877113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.877122 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:16:36.877157 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.877159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.877174 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.877392 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:36.877456 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.878513 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.879611 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.880285 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.880344 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.880399 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.880680 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.880757 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":87,"threads_started":1}
I20260620 23:16:36.880856 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.876514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.16.79.2:36043)
0620 23:16:36.876722 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.876727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.876835 (+ 108us) server_negotiation.cc:407] Connection header received
0620 23:16:36.876926 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.876929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.876946 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.877028 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:36.877358 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.878110 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.879524 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.880261 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.880369 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.880420 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.880533 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.880588 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260620 23:16:36.880883 1728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.876473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.16.79.3:44089)
0620 23:16:36.876643 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.876647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.876923 (+ 276us) server_negotiation.cc:407] Connection header received
0620 23:16:36.876970 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.876972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.876994 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.877063 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:36.877330 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.878444 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.879679 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.880151 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.880236 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.880293 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.880760 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.880805 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260620 23:16:36.880640 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.876405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:36043 (local address 127.0.0.1:44382)
0620 23:16:36.876779 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.876788 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.876839 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.877139 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.877143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.877146 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:36.877250 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.877252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.878223 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.878223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.879430 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.879433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.880375 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.880376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.880448 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.880451 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:36.880503 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 23:16:36.880543 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":26,"thread_start_us":76,"threads_started":1}
I20260620 23:16:36.880755 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.876587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:34447 (local address 127.0.0.1:33490)
0620 23:16:36.876946 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.876955 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.876992 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.877268 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.877269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.877271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:36.877321 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.877323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.878609 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.878610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.879521 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.879523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.880434 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.880435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.880529 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.880533 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:16:36.880585 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 23:16:36.880626 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":75,"threads_started":1}
I20260620 23:16:36.881685 1649 tablet_service.cc:1467] Processing CreateTablet for tablet e1e344e1b206499a95f5ce9131a3738f (DEFAULT_TABLE table=test-table [id=18ae0dd0148244db89308fb19b6046ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:36.881780 1515 tablet_service.cc:1467] Processing CreateTablet for tablet e1e344e1b206499a95f5ce9131a3738f (DEFAULT_TABLE table=test-table [id=18ae0dd0148244db89308fb19b6046ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:36.881927 1380 tablet_service.cc:1467] Processing CreateTablet for tablet e1e344e1b206499a95f5ce9131a3738f (DEFAULT_TABLE table=test-table [id=18ae0dd0148244db89308fb19b6046ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:36.882042 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e344e1b206499a95f5ce9131a3738f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:36.882099 1515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e344e1b206499a95f5ce9131a3738f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:36.882216 1380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e344e1b206499a95f5ce9131a3738f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:36.884850 1732 tablet_bootstrap.cc:492] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420: Bootstrap starting.
I20260620 23:16:36.885504 1732 tablet_bootstrap.cc:654] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:36.885438 1734 tablet_bootstrap.cc:492] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d: Bootstrap starting.
I20260620 23:16:36.886090 1732 log.cc:824] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:36.886323 1734 tablet_bootstrap.cc:654] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:36.886757 1734 log.cc:824] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:36.887041 1733 tablet_bootstrap.cc:492] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213: Bootstrap starting.
I20260620 23:16:36.887451 1734 tablet_bootstrap.cc:492] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d: No bootstrap required, opened a new log
I20260620 23:16:36.887535 1734 ts_tablet_manager.cc:1397] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:16:36.887568 1733 tablet_bootstrap.cc:654] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:36.888012 1733 log.cc:824] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:36.888849 1733 tablet_bootstrap.cc:492] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213: No bootstrap required, opened a new log
I20260620 23:16:36.888907 1732 tablet_bootstrap.cc:492] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420: No bootstrap required, opened a new log
I20260620 23:16:36.888919 1733 ts_tablet_manager.cc:1397] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:16:36.889019 1732 ts_tablet_manager.cc:1397] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:16:36.889312 1734 raft_consensus.cc:348] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5652b9c42cf4fdaae407db6b048b420" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 36043 } } peers { permanent_uuid: "34e15b33a3d5414d8d6cecb43befb62d" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44089 } } peers { permanent_uuid: "76e90fc418484dc3a81fcb43469ec213" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34447 } }
I20260620 23:16:36.889465 1734 raft_consensus.cc:374] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:36.889524 1734 raft_consensus.cc:729] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e15b33a3d5414d8d6cecb43befb62d, State: Initialized, Role: FOLLOWER
I20260620 23:16:36.889643 1734 consensus_queue.cc:260] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5652b9c42cf4fdaae407db6b048b420" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 36043 } } peers { permanent_uuid: "34e15b33a3d5414d8d6cecb43befb62d" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44089 } } peers { permanent_uuid: "76e90fc418484dc3a81fcb43469ec213" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34447 } }
I20260620 23:16:36.889923 1734 ts_tablet_manager.cc:1428] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 23:16:36.890064 1715 heartbeater.cc:503] Master 127.16.79.62:35339 was elected leader, sending a full tablet report...
I20260620 23:16:36.890874 1733 raft_consensus.cc:348] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5652b9c42cf4fdaae407db6b048b420" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 36043 } } peers { permanent_uuid: "34e15b33a3d5414d8d6cecb43befb62d" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44089 } } peers { permanent_uuid: "76e90fc418484dc3a81fcb43469ec213" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34447 } }
I20260620 23:16:36.890901 1732 raft_consensus.cc:348] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5652b9c42cf4fdaae407db6b048b420" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 36043 } } peers { permanent_uuid: "34e15b33a3d5414d8d6cecb43befb62d" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44089 } } peers { permanent_uuid: "76e90fc418484dc3a81fcb43469ec213" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34447 } }
I20260620 23:16:36.891043 1732 raft_consensus.cc:374] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:36.891043 1733 raft_consensus.cc:374] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:36.891086 1733 raft_consensus.cc:729] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e90fc418484dc3a81fcb43469ec213, State: Initialized, Role: FOLLOWER
I20260620 23:16:36.891114 1732 raft_consensus.cc:729] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5652b9c42cf4fdaae407db6b048b420, State: Initialized, Role: FOLLOWER
I20260620 23:16:36.891204 1733 consensus_queue.cc:260] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5652b9c42cf4fdaae407db6b048b420" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 36043 } } peers { permanent_uuid: "34e15b33a3d5414d8d6cecb43befb62d" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44089 } } peers { permanent_uuid: "76e90fc418484dc3a81fcb43469ec213" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34447 } }
I20260620 23:16:36.891249 1732 consensus_queue.cc:260] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5652b9c42cf4fdaae407db6b048b420" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 36043 } } peers { permanent_uuid: "34e15b33a3d5414d8d6cecb43befb62d" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44089 } } peers { permanent_uuid: "76e90fc418484dc3a81fcb43469ec213" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34447 } }
I20260620 23:16:36.891511 1733 ts_tablet_manager.cc:1428] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:16:36.891511 1732 ts_tablet_manager.cc:1428] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:16:36.891530 1581 heartbeater.cc:503] Master 127.16.79.62:35339 was elected leader, sending a full tablet report...
I20260620 23:16:36.891641 1446 heartbeater.cc:503] Master 127.16.79.62:35339 was elected leader, sending a full tablet report...
W20260620 23:16:36.919059 1582 tablet.cc:2369] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:16:36.995857 1740 raft_consensus.cc:482] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:16:36.995982 1740 raft_consensus.cc:504] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5652b9c42cf4fdaae407db6b048b420" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 36043 } } peers { permanent_uuid: "34e15b33a3d5414d8d6cecb43befb62d" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44089 } } peers { permanent_uuid: "76e90fc418484dc3a81fcb43469ec213" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34447 } }
I20260620 23:16:36.996512 1740 leader_election.cc:288] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5652b9c42cf4fdaae407db6b048b420 (127.16.79.2:36043), 34e15b33a3d5414d8d6cecb43befb62d (127.16.79.3:44089)
I20260620 23:16:37.000576 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.996797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:51139 (local address 127.16.79.2:36043)
0620 23:16:36.996924 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.996926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.996934 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:16:36.996962 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.996963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.996976 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.997025 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:36.997230 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.997997 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.999597 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.000199 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.000277 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.000287 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.000471 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.000511 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 23:16:37.000576 1728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.996853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:53463 (local address 127.16.79.3:44089)
0620 23:16:36.996896 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.996898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.996993 (+ 95us) server_negotiation.cc:407] Connection header received
0620 23:16:36.997096 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.997097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.997111 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.997173 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:36.997318 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.997996 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.999597 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.000199 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.000277 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.000287 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.000471 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.000507 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:16:37.000588 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.996728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:36043 (local address 127.16.79.1:51139)
0620 23:16:36.996789 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.996800 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.996841 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.997079 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.997079 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.997081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:36.997168 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.997170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.998104 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.998106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.999484 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.999487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.000295 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.000296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.000386 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.000388 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:16:37.000447 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 23:16:37.000496 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":71}
I20260620 23:16:37.000595 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.996728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:44089 (local address 127.16.79.1:53463)
0620 23:16:36.996933 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.996943 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.997017 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.997181 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.997181 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.997183 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:36.997245 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.997246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.998104 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.998106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.999484 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.999487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.000296 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.000296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.000379 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.000384 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:16:37.000447 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:16:37.000496 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":211,"thread_start_us":62,"threads_started":1}
I20260620 23:16:37.001029 1535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e344e1b206499a95f5ce9131a3738f" candidate_uuid: "76e90fc418484dc3a81fcb43469ec213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5652b9c42cf4fdaae407db6b048b420" is_pre_election: true
I20260620 23:16:37.001180 1535 raft_consensus.cc:2454] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76e90fc418484dc3a81fcb43469ec213 in term 0.
I20260620 23:16:37.001264 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e344e1b206499a95f5ce9131a3738f" candidate_uuid: "76e90fc418484dc3a81fcb43469ec213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e15b33a3d5414d8d6cecb43befb62d" is_pre_election: true
I20260620 23:16:37.001412 1669 raft_consensus.cc:2454] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76e90fc418484dc3a81fcb43469ec213 in term 0.
I20260620 23:16:37.001403 1336 leader_election.cc:302] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76e90fc418484dc3a81fcb43469ec213, c5652b9c42cf4fdaae407db6b048b420; no voters:
I20260620 23:16:37.001514 1740 raft_consensus.cc:2781] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:16:37.001565 1740 raft_consensus.cc:482] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:16:37.001619 1740 raft_consensus.cc:3037] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:37.002416 1740 raft_consensus.cc:504] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5652b9c42cf4fdaae407db6b048b420" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 36043 } } peers { permanent_uuid: "34e15b33a3d5414d8d6cecb43befb62d" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44089 } } peers { permanent_uuid: "76e90fc418484dc3a81fcb43469ec213" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34447 } }
I20260620 23:16:37.002607 1740 leader_election.cc:288] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [CANDIDATE]: Term 1 election: Requested vote from peers c5652b9c42cf4fdaae407db6b048b420 (127.16.79.2:36043), 34e15b33a3d5414d8d6cecb43befb62d (127.16.79.3:44089)
I20260620 23:16:37.002712 1535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e344e1b206499a95f5ce9131a3738f" candidate_uuid: "76e90fc418484dc3a81fcb43469ec213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5652b9c42cf4fdaae407db6b048b420"
I20260620 23:16:37.002682 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e344e1b206499a95f5ce9131a3738f" candidate_uuid: "76e90fc418484dc3a81fcb43469ec213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e15b33a3d5414d8d6cecb43befb62d"
I20260620 23:16:37.002786 1669 raft_consensus.cc:3037] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:37.002786 1535 raft_consensus.cc:3037] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:37.003494 1535 raft_consensus.cc:2454] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76e90fc418484dc3a81fcb43469ec213 in term 1.
I20260620 23:16:37.003594 1669 raft_consensus.cc:2454] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76e90fc418484dc3a81fcb43469ec213 in term 1.
I20260620 23:16:37.003840 1335 leader_election.cc:302] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e15b33a3d5414d8d6cecb43befb62d, 76e90fc418484dc3a81fcb43469ec213; no voters:
I20260620 23:16:37.003952 1740 raft_consensus.cc:2781] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:37.004153 1740 raft_consensus.cc:686] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 1 LEADER]: Becoming Leader. State: Replica: 76e90fc418484dc3a81fcb43469ec213, State: Running, Role: LEADER
I20260620 23:16:37.004271 1740 consensus_queue.cc:237] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5652b9c42cf4fdaae407db6b048b420" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 36043 } } peers { permanent_uuid: "34e15b33a3d5414d8d6cecb43befb62d" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44089 } } peers { permanent_uuid: "76e90fc418484dc3a81fcb43469ec213" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34447 } }
I20260620 23:16:37.005100 1253 catalog_manager.cc:5495] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76e90fc418484dc3a81fcb43469ec213 (127.16.79.1). New cstate: current_term: 1 leader_uuid: "76e90fc418484dc3a81fcb43469ec213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5652b9c42cf4fdaae407db6b048b420" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 36043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e15b33a3d5414d8d6cecb43befb62d" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76e90fc418484dc3a81fcb43469ec213" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 34447 } health_report { overall_health: HEALTHY } } }
W20260620 23:16:37.010087 1447 tablet.cc:2369] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:16:37.031232 1731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.028432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.16.79.1:34447)
0620 23:16:37.028481 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.028482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.028490 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:16:37.028521 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.028522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.028539 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.028609 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:37.028801 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.029243 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.030219 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.030409 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.030476 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.030773 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:37.030897 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:37.030978 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.031092 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.031178 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:16:37.033305 1669 raft_consensus.cc:1261] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d [term 1 FOLLOWER]: Refusing update from remote peer 76e90fc418484dc3a81fcb43469ec213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:37.033317 1535 raft_consensus.cc:1261] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420 [term 1 FOLLOWER]: Refusing update from remote peer 76e90fc418484dc3a81fcb43469ec213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:37.033764 1740 consensus_queue.cc:1035] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e15b33a3d5414d8d6cecb43befb62d" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:37.034013 1742 consensus_queue.cc:1035] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5652b9c42cf4fdaae407db6b048b420" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 36043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:37.036957 1746 mvcc.cc:205] Tried to move back new op lower bound from 7299061338244378624 to 7299061338130268160. Current Snapshot: MvccSnapshot[applied={T|T < 7299061338244378624}]
I20260620 23:16:37.039096 1748 mvcc.cc:205] Tried to move back new op lower bound from 7299061338244378624 to 7299061338130268160. Current Snapshot: MvccSnapshot[applied={T|T < 7299061338244378624}]
I20260620 23:16:37.039466 1253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:16:37.039567 1253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:16:37.040305 1253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17a9406aa6ce476ba91d2ccac8785154: Sending DeleteTablet for 3 replicas of tablet e1e344e1b206499a95f5ce9131a3738f
I20260620 23:16:37.040781 1515 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e344e1b206499a95f5ce9131a3738f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:37 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:44382
I20260620 23:16:37.040868 1380 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e344e1b206499a95f5ce9131a3738f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:37 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:33490
I20260620 23:16:37.041080 1756 tablet_replica.cc:330] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213: stopping tablet replica
I20260620 23:16:37.041056 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e344e1b206499a95f5ce9131a3738f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:37 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:44926
I20260620 23:16:37.041232 1755 tablet_replica.cc:330] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420: stopping tablet replica
I20260620 23:16:37.041241 1756 raft_consensus.cc:2229] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:16:37.041338 1756 raft_consensus.cc:2258] T e1e344e1b206499a95f5ce9131a3738f P 76e90fc418484dc3a81fcb43469ec213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:37.041361 1755 raft_consensus.cc:2229] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:16:37.041381 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 1317
I20260620 23:16:37.041540 1755 raft_consensus.cc:2258] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:37.042083 1757 tablet_replica.cc:330] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d: stopping tablet replica
I20260620 23:16:37.042253 1757 raft_consensus.cc:2229] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:16:37.042402 1757 raft_consensus.cc:2258] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:37.042662 1755 ts_tablet_manager.cc:1905] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:16:37.042788 1757 ts_tablet_manager.cc:1905] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:16:37.044672 1757 ts_tablet_manager.cc:1918] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:16:37.044672 1755 ts_tablet_manager.cc:1918] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:16:37.044768 1757 log.cc:1196] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/e1e344e1b206499a95f5ce9131a3738f
I20260620 23:16:37.044785 1755 log.cc:1196] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/e1e344e1b206499a95f5ce9131a3738f
I20260620 23:16:37.045068 1755 ts_tablet_manager.cc:1939] T e1e344e1b206499a95f5ce9131a3738f P c5652b9c42cf4fdaae407db6b048b420: Deleting consensus metadata
I20260620 23:16:37.045068 1757 ts_tablet_manager.cc:1939] T e1e344e1b206499a95f5ce9131a3738f P 34e15b33a3d5414d8d6cecb43befb62d: Deleting consensus metadata
I20260620 23:16:37.045475 1241 catalog_manager.cc:4841] TS 34e15b33a3d5414d8d6cecb43befb62d (127.16.79.3:44089): tablet e1e344e1b206499a95f5ce9131a3738f (table test-table [id=18ae0dd0148244db89308fb19b6046ff]) successfully deleted
I20260620 23:16:37.045521 1242 catalog_manager.cc:4841] TS c5652b9c42cf4fdaae407db6b048b420 (127.16.79.2:36043): tablet e1e344e1b206499a95f5ce9131a3738f (table test-table [id=18ae0dd0148244db89308fb19b6046ff]) successfully deleted
W20260620 23:16:37.047590 1241 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:34447 (error 108)
I20260620 23:16:37.047863 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 1451
I20260620 23:16:37.048048 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.047945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:34447 (local address 127.0.0.1:33508)
0620 23:16:37.047988 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:37.048015 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 23:16:37.048148 1241 catalog_manager.cc:4584] TS 76e90fc418484dc3a81fcb43469ec213 (127.16.79.1:34447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1e344e1b206499a95f5ce9131a3738f: Network error: Client connection negotiation failed: client connection to 127.16.79.1:34447: connect: Connection refused (error 111)
I20260620 23:16:37.052969 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 1586
I20260620 23:16:37.057395 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 1223
2026-06-20T23:16:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2066 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T23:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:37Z Disabled control of system clock
I20260620 23:16:37.084653 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45093
--webserver_interface=127.16.79.62
--webserver_port=0
--builtin_ntp_servers=127.16.79.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:45093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:16:37.163597 1762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:37.163798 1762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:37.163818 1762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:37.164887 1762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:37.164929 1762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:37.164943 1762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:37.164955 1762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:37.164968 1762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:37.166196 1762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:45093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:37.166376 1762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:37.166592 1762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:37.171654 1768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:37.171732 1767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:37.171759 1770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:37.171859 1762 server_base.cc:1034] running on GCE node
I20260620 23:16:37.172274 1762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:37.172509 1762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:37.173663 1762 hybrid_clock.cc:648] HybridClock initialized: now 1781997397173627 us; error 50 us; skew 500 ppm
I20260620 23:16:37.174914 1762 webserver.cc:460] Webserver started at http://127.16.79.62:34935/ using document root <none> and password file <none>
I20260620 23:16:37.175158 1762 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:37.175205 1762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:37.175340 1762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:37.176373 1762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "ff98bf4fdc5048629568e2e0c6ef946b"
format_stamp: "Formatted at 2026-06-20 23:16:37 on dist-test-slave-pswb"
server_key: "87afc0dbc094edbdeff306fb97236db9"
server_key_iv: "0cfdb59374f794660ec108bab1d70a40"
server_key_version: "encryptionkey@0"
I20260620 23:16:37.176798 1762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "ff98bf4fdc5048629568e2e0c6ef946b"
format_stamp: "Formatted at 2026-06-20 23:16:37 on dist-test-slave-pswb"
server_key: "87afc0dbc094edbdeff306fb97236db9"
server_key_iv: "0cfdb59374f794660ec108bab1d70a40"
server_key_version: "encryptionkey@0"
I20260620 23:16:37.178205 1762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:16:37.178992 1776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:37.179171 1762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:16:37.179281 1762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "ff98bf4fdc5048629568e2e0c6ef946b"
format_stamp: "Formatted at 2026-06-20 23:16:37 on dist-test-slave-pswb"
server_key: "87afc0dbc094edbdeff306fb97236db9"
server_key_iv: "0cfdb59374f794660ec108bab1d70a40"
server_key_version: "encryptionkey@0"
I20260620 23:16:37.179368 1762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:37.188506 1762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:37.188840 1762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:37.188992 1762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:37.192900 1762 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:45093
I20260620 23:16:37.192932 1828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:45093 every 8 connection(s)
I20260620 23:16:37.193248 1762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:37.194622 1829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:37.198127 1829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b: Bootstrap starting.
I20260620 23:16:37.198575 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 1762
I20260620 23:16:37.198706 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:37.198827 1829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:37.198926 16700 external_mini_cluster.cc:1428] Setting key ad85eaf1eabec797c5d92cd1bd094793
I20260620 23:16:37.199165 1829 log.cc:824] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:37.200026 1829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b: No bootstrap required, opened a new log
I20260620 23:16:37.202481 1829 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff98bf4fdc5048629568e2e0c6ef946b" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45093 } }
I20260620 23:16:37.202633 1829 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:37.202677 1829 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff98bf4fdc5048629568e2e0c6ef946b, State: Initialized, Role: FOLLOWER
I20260620 23:16:37.202795 1829 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff98bf4fdc5048629568e2e0c6ef946b" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45093 } }
I20260620 23:16:37.202889 1829 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:37.202929 1829 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:37.202977 1829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:37.203825 1829 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff98bf4fdc5048629568e2e0c6ef946b" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45093 } }
I20260620 23:16:37.203948 1829 leader_election.cc:302] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff98bf4fdc5048629568e2e0c6ef946b; no voters:
I20260620 23:16:37.204109 1829 leader_election.cc:288] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:37.204247 1834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:37.204435 1834 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [term 1 LEADER]: Becoming Leader. State: Replica: ff98bf4fdc5048629568e2e0c6ef946b, State: Running, Role: LEADER
I20260620 23:16:37.204463 1829 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:37.204561 1834 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff98bf4fdc5048629568e2e0c6ef946b" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45093 } }
I20260620 23:16:37.205258 1835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff98bf4fdc5048629568e2e0c6ef946b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff98bf4fdc5048629568e2e0c6ef946b" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45093 } } }
I20260620 23:16:37.205348 1835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:37.205504 1836 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff98bf4fdc5048629568e2e0c6ef946b. Latest consensus state: current_term: 1 leader_uuid: "ff98bf4fdc5048629568e2e0c6ef946b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff98bf4fdc5048629568e2e0c6ef946b" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45093 } } }
I20260620 23:16:37.205536 1843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:37.205566 1836 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:37.205972 1843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:37.206247 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.199410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.16.79.62:45093)
0620 23:16:37.199603 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.199613 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.199641 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:16:37.200234 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.200242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.200310 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.200462 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:37.200665 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.202112 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.203505 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.204234 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.204818 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.205206 (+ 388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:37.205211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:37.205211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:37.205448 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:37.205488 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:37.205503 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:37.205873 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.205914 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.206003 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260620 23:16:37.207855 1843 catalog_manager.cc:1347] Generated new cluster ID: 3832bd3380134675a78e188014ff3419
I20260620 23:16:37.207909 1843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:37.235684 1843 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:37.236900 1843 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:37.247301 1843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b: Generated new TSK 0
I20260620 23:16:37.247478 1843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:37.251791 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45093
--builtin_ntp_servers=127.16.79.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:16:37.336313 1853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:37.336611 1853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:37.336659 1853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:37.338092 1853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:37.338168 1853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:37.338218 1853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:16:37.339798 1853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:37.340116 1853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:37.340457 1853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:37.346179 1859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:37.346271 1858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:37.346319 1861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:37.346598 1853 server_base.cc:1034] running on GCE node
I20260620 23:16:37.346895 1853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:37.347203 1853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:37.348376 1853 hybrid_clock.cc:648] HybridClock initialized: now 1781997397348364 us; error 29 us; skew 500 ppm
I20260620 23:16:37.349697 1853 webserver.cc:460] Webserver started at http://127.16.79.1:41919/ using document root <none> and password file <none>
I20260620 23:16:37.349901 1853 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:37.349946 1853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:37.350039 1853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:37.351116 1853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "19bb3b2c268b4d3495046015ea6685a8"
format_stamp: "Formatted at 2026-06-20 23:16:37 on dist-test-slave-pswb"
server_key: "359812694c76586e11ab525ec1a4c17c"
server_key_iv: "f12cabaf260527a64a08c7745d2a3e01"
server_key_version: "encryptionkey@0"
I20260620 23:16:37.351437 1853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "19bb3b2c268b4d3495046015ea6685a8"
format_stamp: "Formatted at 2026-06-20 23:16:37 on dist-test-slave-pswb"
server_key: "359812694c76586e11ab525ec1a4c17c"
server_key_iv: "f12cabaf260527a64a08c7745d2a3e01"
server_key_version: "encryptionkey@0"
I20260620 23:16:37.352792 1853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:16:37.353510 1867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:37.353693 1853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:16:37.353788 1853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "19bb3b2c268b4d3495046015ea6685a8"
format_stamp: "Formatted at 2026-06-20 23:16:37 on dist-test-slave-pswb"
server_key: "359812694c76586e11ab525ec1a4c17c"
server_key_iv: "f12cabaf260527a64a08c7745d2a3e01"
server_key_version: "encryptionkey@0"
I20260620 23:16:37.353864 1853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:37.368855 1853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:37.369110 1853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:37.369252 1853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:37.369489 1853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:37.369779 1853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:37.369831 1853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:37.369870 1853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:37.369897 1853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:37.376134 1853 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:40457
I20260620 23:16:37.376174 1980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:40457 every 8 connection(s)
I20260620 23:16:37.376473 1853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:16:37.377676 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 1853
I20260620 23:16:37.377796 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:16:37.378021 16700 external_mini_cluster.cc:1428] Setting key 1fb23843665c72443b817874eb8eeb56
I20260620 23:16:37.379091 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45093
--builtin_ntp_servers=127.16.79.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:16:37.382284 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.377186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:58631 (local address 127.16.79.62:45093)
0620 23:16:37.377248 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.377250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.377476 (+ 226us) server_negotiation.cc:407] Connection header received
0620 23:16:37.378090 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.378091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.378104 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.378154 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:37.378916 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.379515 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.380811 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.381002 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.381065 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.381936 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:37.381938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:37.381938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:37.381943 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:37.381958 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:37.381962 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:37.382009 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.382141 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.382213 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:16:37.382551 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.377180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45093 (local address 127.16.79.1:58631)
0620 23:16:37.377367 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:37.377397 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:37.377976 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:37.378216 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:37.378220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:37.378229 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:37.378823 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.378829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.379607 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.379608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.380671 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.380674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.381096 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.381097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.381248 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.381794 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:16:37.381794 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:37.381830 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:16:37.381856 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:37.382028 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:37.382033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:37.382036 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:37.382109 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:16:37.382258 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260620 23:16:37.383059 1981 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45093
I20260620 23:16:37.383158 1981 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:37.383389 1981 heartbeater.cc:511] Master 127.16.79.62:45093 requested a full tablet report, sending...
I20260620 23:16:37.383865 1793 ts_manager.cc:194] Registered new tserver with Master: 19bb3b2c268b4d3495046015ea6685a8 (127.16.79.1:40457)
I20260620 23:16:37.384474 1793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.79.1:58631
W20260620 23:16:37.460291 1984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:37.460479 1984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:37.460500 1984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:37.461627 1984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:37.461665 1984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:37.461690 1984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:16:37.462980 1984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:37.463217 1984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:37.463493 1984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:37.468370 1989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:37.468499 1992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:37.468432 1990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:37.468909 1984 server_base.cc:1034] running on GCE node
I20260620 23:16:37.469146 1984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:37.469375 1984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:37.470523 1984 hybrid_clock.cc:648] HybridClock initialized: now 1781997397470512 us; error 26 us; skew 500 ppm
I20260620 23:16:37.471884 1984 webserver.cc:460] Webserver started at http://127.16.79.2:40945/ using document root <none> and password file <none>
I20260620 23:16:37.472126 1984 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:37.472177 1984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:37.472330 1984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:37.473414 1984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "19969717f9ab42a7bf40f70b9065561d"
format_stamp: "Formatted at 2026-06-20 23:16:37 on dist-test-slave-pswb"
server_key: "54d013cafec224a7dc2c912b86b7936c"
server_key_iv: "6bab4436085a5020c935f8afb5625aad"
server_key_version: "encryptionkey@0"
I20260620 23:16:37.473750 1984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "19969717f9ab42a7bf40f70b9065561d"
format_stamp: "Formatted at 2026-06-20 23:16:37 on dist-test-slave-pswb"
server_key: "54d013cafec224a7dc2c912b86b7936c"
server_key_iv: "6bab4436085a5020c935f8afb5625aad"
server_key_version: "encryptionkey@0"
I20260620 23:16:37.475195 1984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:16:37.476017 1998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:37.476354 1984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:37.476531 1984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "19969717f9ab42a7bf40f70b9065561d"
format_stamp: "Formatted at 2026-06-20 23:16:37 on dist-test-slave-pswb"
server_key: "54d013cafec224a7dc2c912b86b7936c"
server_key_iv: "6bab4436085a5020c935f8afb5625aad"
server_key_version: "encryptionkey@0"
I20260620 23:16:37.476611 1984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:37.483421 1984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:37.483671 1984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:37.483814 1984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:37.484017 1984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:37.484318 1984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:37.484371 1984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:37.484403 1984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:37.484431 1984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:37.489918 1984 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:35737
I20260620 23:16:37.489988 2111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:35737 every 8 connection(s)
I20260620 23:16:37.490286 1984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:16:37.494485 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 1984
I20260620 23:16:37.494617 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:16:37.494894 16700 external_mini_cluster.cc:1428] Setting key 7efa39e0d4e80e8df606bb01ac9db946
I20260620 23:16:37.496133 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.490994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:56999 (local address 127.16.79.62:45093)
0620 23:16:37.491040 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.491042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.491269 (+ 227us) server_negotiation.cc:407] Connection header received
0620 23:16:37.491678 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.491679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.491689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.491741 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:37.492314 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.492871 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.494409 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.494619 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.494687 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.495866 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:37.495867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:37.495867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:37.495872 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:37.495888 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:37.495896 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:37.495957 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.496037 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.496060 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:16:37.496264 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45093
--builtin_ntp_servers=127.16.79.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:16:37.496371 2114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.490983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45093 (local address 127.16.79.2:56999)
0620 23:16:37.491160 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:37.491189 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:37.491604 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:37.491793 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:37.491796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:37.491803 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:37.492251 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.492254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.493056 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.493058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.494251 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.494256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.494710 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.494712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.494861 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.495679 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:16:37.495679 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:37.495713 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:16:37.495753 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:37.495959 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:37.495965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:37.495968 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:37.496038 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 23:16:37.496144 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260620 23:16:37.496811 2112 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45093
I20260620 23:16:37.496908 2112 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:37.497087 2112 heartbeater.cc:511] Master 127.16.79.62:45093 requested a full tablet report, sending...
I20260620 23:16:37.497462 1793 ts_manager.cc:194] Registered new tserver with Master: 19969717f9ab42a7bf40f70b9065561d (127.16.79.2:35737)
I20260620 23:16:37.497823 1793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.79.2:56999
W20260620 23:16:37.578773 2115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:37.579113 2115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:37.579181 2115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:37.581205 2115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:37.581274 2115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:37.581317 2115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:16:37.583153 2115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:45093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:37.583387 2115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:37.583652 2115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:37.588522 2120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:37.588506 2121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:37.588838 2115 server_base.cc:1034] running on GCE node
W20260620 23:16:37.588522 2123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:37.589112 2115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:37.589300 2115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:37.590456 2115 hybrid_clock.cc:648] HybridClock initialized: now 1781997397590444 us; error 27 us; skew 500 ppm
I20260620 23:16:37.591776 2115 webserver.cc:460] Webserver started at http://127.16.79.3:37295/ using document root <none> and password file <none>
I20260620 23:16:37.592010 2115 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:37.592077 2115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:37.592227 2115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:37.593369 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "29d4ef3983f645b9a7f12ce7507dd3ed"
format_stamp: "Formatted at 2026-06-20 23:16:37 on dist-test-slave-pswb"
server_key: "ad9931d218194c71fd77906c5211e364"
server_key_iv: "a0185f78921ac2fa1ba57e631111b43a"
server_key_version: "encryptionkey@0"
I20260620 23:16:37.593717 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "29d4ef3983f645b9a7f12ce7507dd3ed"
format_stamp: "Formatted at 2026-06-20 23:16:37 on dist-test-slave-pswb"
server_key: "ad9931d218194c71fd77906c5211e364"
server_key_iv: "a0185f78921ac2fa1ba57e631111b43a"
server_key_version: "encryptionkey@0"
I20260620 23:16:37.594992 2115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:16:37.595744 2129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:37.595945 2115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:37.596050 2115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "29d4ef3983f645b9a7f12ce7507dd3ed"
format_stamp: "Formatted at 2026-06-20 23:16:37 on dist-test-slave-pswb"
server_key: "ad9931d218194c71fd77906c5211e364"
server_key_iv: "a0185f78921ac2fa1ba57e631111b43a"
server_key_version: "encryptionkey@0"
I20260620 23:16:37.596122 2115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:37.602694 2115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:37.602946 2115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:37.603087 2115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:37.603289 2115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:37.603633 2115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:37.603693 2115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:37.603724 2115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:37.603751 2115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:37.610309 2115 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:43397
I20260620 23:16:37.610360 2243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:43397 every 8 connection(s)
I20260620 23:16:37.610668 2115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:16:37.611646 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 2115
I20260620 23:16:37.611738 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:16:37.611932 16700 external_mini_cluster.cc:1428] Setting key 87b31bf83233665bd75dba46783bc94e
I20260620 23:16:37.616478 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.611388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:41937 (local address 127.16.79.62:45093)
0620 23:16:37.611461 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.611463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.611707 (+ 244us) server_negotiation.cc:407] Connection header received
0620 23:16:37.612313 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.612314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.612372 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.612430 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:37.613281 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.613752 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.614857 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.615039 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.615100 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.615989 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:37.615991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:37.615992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:37.615999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:37.616030 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:37.616037 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:37.616124 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.616396 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.616421 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:16:37.616894 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.611371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:45093 (local address 127.16.79.3:41937)
0620 23:16:37.611569 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:37.611621 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:37.612182 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:37.612517 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:37.612521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:37.612530 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:37.613204 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.613209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.613990 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.613991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.614739 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.614741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.615126 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.615127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.615229 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.615784 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:16:37.615785 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:37.615811 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:16:37.615831 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:37.616205 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:37.616216 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:37.616221 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:16:37.616366 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 23:16:37.616571 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260620 23:16:37.617507 2244 heartbeater.cc:347] Connected to a master server at 127.16.79.62:45093
I20260620 23:16:37.617614 2244 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:37.617833 2244 heartbeater.cc:511] Master 127.16.79.62:45093 requested a full tablet report, sending...
I20260620 23:16:37.618309 1793 ts_manager.cc:194] Registered new tserver with Master: 29d4ef3983f645b9a7f12ce7507dd3ed (127.16.79.3:43397)
I20260620 23:16:37.618842 1793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.79.3:41937
I20260620 23:16:37.624032 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:16:37.628525 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.625464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.16.79.62:45093)
0620 23:16:37.625570 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.625573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.625586 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:16:37.625616 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.625617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.625626 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.625684 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:37.625893 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.626337 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.627111 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.627283 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.627342 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.628292 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:37.628293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:37.628294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:37.628298 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:37.628317 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:37.628321 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:37.628381 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.628434 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.628460 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:16:37.630561 1793 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:16:37.632121 1793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:16:37.638060 2252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.634403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:35737 (local address 127.0.0.1:33318)
0620 23:16:37.634529 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:37.634564 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:37.634638 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:37.634901 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:37.634903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:37.634905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:37.634967 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.634968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.635993 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.635994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.636904 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.636906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.637786 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.637786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.637860 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.637872 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 23:16:37.637937 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 23:16:37.637987 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
I20260620 23:16:37.638069 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.634495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.16.79.2:35737)
0620 23:16:37.634664 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.634668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.634679 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:16:37.634722 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.634724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.634740 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.634834 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:37.635052 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.635911 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.636987 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.637686 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.637785 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.637870 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.637946 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.637985 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260620 23:16:37.638917 2254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.634618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:40457 (local address 127.0.0.1:53828)
0620 23:16:37.634764 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:37.634775 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:37.634817 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:37.635329 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:37.635330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:37.635332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:37.635371 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.635372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.636717 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.636718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.637668 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.637670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.638704 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.638705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.638762 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.638765 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:37.638813 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 23:16:37.638850 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":33,"thread_start_us":79,"threads_started":1}
I20260620 23:16:37.638931 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.634998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.16.79.1:40457)
0620 23:16:37.635154 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.635158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.635167 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:16:37.635205 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.635207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.635223 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.635481 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:37.635533 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.636598 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.637750 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.638605 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.638701 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.638780 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.638825 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.638859 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260620 23:16:37.638916 2046 tablet_service.cc:1467] Processing CreateTablet for tablet 4a4ec58e199e4155a80d61f8699d6ec1 (DEFAULT_TABLE table=test-table [id=6855a7c4e5e14ec6be3752c68e8ffb2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:37.639176 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.634874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.16.79.3:43397)
0620 23:16:37.635213 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.635217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.635226 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:16:37.635258 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.635260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.635274 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.635341 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:37.635518 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.636608 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.637911 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.638608 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.638682 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.638754 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.639051 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.639091 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":66,"threads_started":1}
I20260620 23:16:37.639266 2046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4ec58e199e4155a80d61f8699d6ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:37.639405 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.634798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:43397 (local address 127.0.0.1:35616)
0620 23:16:37.634989 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:37.635000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:37.635067 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:37.635398 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:37.635399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:37.635402 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:37.635443 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.635445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.636743 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.636743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.637827 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.637830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.638790 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.638791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.638855 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.638860 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:16:37.638910 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 23:16:37.638952 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260620 23:16:37.639572 1915 tablet_service.cc:1467] Processing CreateTablet for tablet 4a4ec58e199e4155a80d61f8699d6ec1 (DEFAULT_TABLE table=test-table [id=6855a7c4e5e14ec6be3752c68e8ffb2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:37.639880 1915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4ec58e199e4155a80d61f8699d6ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:37.640069 2178 tablet_service.cc:1467] Processing CreateTablet for tablet 4a4ec58e199e4155a80d61f8699d6ec1 (DEFAULT_TABLE table=test-table [id=6855a7c4e5e14ec6be3752c68e8ffb2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:37.640380 2178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4ec58e199e4155a80d61f8699d6ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:37.642509 2258 tablet_bootstrap.cc:492] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8: Bootstrap starting.
I20260620 23:16:37.642758 2259 tablet_bootstrap.cc:492] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d: Bootstrap starting.
I20260620 23:16:37.643163 2258 tablet_bootstrap.cc:654] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:37.643411 2259 tablet_bootstrap.cc:654] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:37.643560 2260 tablet_bootstrap.cc:492] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed: Bootstrap starting.
I20260620 23:16:37.643484 2258 log.cc:824] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:37.643829 2259 log.cc:824] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:37.644237 2260 tablet_bootstrap.cc:654] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:37.644340 2258 tablet_bootstrap.cc:492] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8: No bootstrap required, opened a new log
I20260620 23:16:37.644410 2258 ts_tablet_manager.cc:1397] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:16:37.644531 2259 tablet_bootstrap.cc:492] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d: No bootstrap required, opened a new log
I20260620 23:16:37.644533 2260 log.cc:824] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:37.644588 2259 ts_tablet_manager.cc:1397] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:16:37.645157 2260 tablet_bootstrap.cc:492] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed: No bootstrap required, opened a new log
I20260620 23:16:37.645236 2260 ts_tablet_manager.cc:1397] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 23:16:37.647032 2258 raft_consensus.cc:348] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19969717f9ab42a7bf40f70b9065561d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35737 } } peers { permanent_uuid: "19bb3b2c268b4d3495046015ea6685a8" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40457 } } peers { permanent_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43397 } }
I20260620 23:16:37.647092 2259 raft_consensus.cc:348] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19969717f9ab42a7bf40f70b9065561d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35737 } } peers { permanent_uuid: "19bb3b2c268b4d3495046015ea6685a8" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40457 } } peers { permanent_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43397 } }
I20260620 23:16:37.647208 2259 raft_consensus.cc:374] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:37.647207 2258 raft_consensus.cc:374] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:37.647115 2260 raft_consensus.cc:348] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19969717f9ab42a7bf40f70b9065561d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35737 } } peers { permanent_uuid: "19bb3b2c268b4d3495046015ea6685a8" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40457 } } peers { permanent_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43397 } }
I20260620 23:16:37.647243 2259 raft_consensus.cc:729] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19969717f9ab42a7bf40f70b9065561d, State: Initialized, Role: FOLLOWER
I20260620 23:16:37.647243 2258 raft_consensus.cc:729] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bb3b2c268b4d3495046015ea6685a8, State: Initialized, Role: FOLLOWER
I20260620 23:16:37.647300 2260 raft_consensus.cc:374] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:37.647364 2258 consensus_queue.cc:260] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19969717f9ab42a7bf40f70b9065561d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35737 } } peers { permanent_uuid: "19bb3b2c268b4d3495046015ea6685a8" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40457 } } peers { permanent_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43397 } }
I20260620 23:16:37.647356 2259 consensus_queue.cc:260] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19969717f9ab42a7bf40f70b9065561d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35737 } } peers { permanent_uuid: "19bb3b2c268b4d3495046015ea6685a8" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40457 } } peers { permanent_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43397 } }
I20260620 23:16:37.647394 2260 raft_consensus.cc:729] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d4ef3983f645b9a7f12ce7507dd3ed, State: Initialized, Role: FOLLOWER
I20260620 23:16:37.647522 2260 consensus_queue.cc:260] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19969717f9ab42a7bf40f70b9065561d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35737 } } peers { permanent_uuid: "19bb3b2c268b4d3495046015ea6685a8" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40457 } } peers { permanent_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43397 } }
I20260620 23:16:37.647619 2259 ts_tablet_manager.cc:1428] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:16:37.647639 1981 heartbeater.cc:503] Master 127.16.79.62:45093 was elected leader, sending a full tablet report...
I20260620 23:16:37.647742 2112 heartbeater.cc:503] Master 127.16.79.62:45093 was elected leader, sending a full tablet report...
I20260620 23:16:37.647822 2260 ts_tablet_manager.cc:1428] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:16:37.647948 2244 heartbeater.cc:503] Master 127.16.79.62:45093 was elected leader, sending a full tablet report...
I20260620 23:16:37.647608 2258 ts_tablet_manager.cc:1428] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20260620 23:16:37.740895 2113 tablet.cc:2369] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:16:37.861234 2245 tablet.cc:2369] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:16:37.877225 1982 tablet.cc:2369] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:16:37.963493 2264 raft_consensus.cc:482] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:16:37.963609 2264 raft_consensus.cc:504] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19969717f9ab42a7bf40f70b9065561d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35737 } } peers { permanent_uuid: "19bb3b2c268b4d3495046015ea6685a8" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40457 } } peers { permanent_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43397 } }
I20260620 23:16:37.963922 2264 leader_election.cc:288] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19bb3b2c268b4d3495046015ea6685a8 (127.16.79.1:40457), 29d4ef3983f645b9a7f12ce7507dd3ed (127.16.79.3:43397)
I20260620 23:16:37.966852 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.964332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:43997 (local address 127.16.79.3:43397)
0620 23:16:37.964381 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.964384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.964484 (+ 100us) server_negotiation.cc:407] Connection header received
0620 23:16:37.964518 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.964519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.964530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.964587 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:37.964760 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.965264 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.966151 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.966515 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.966569 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.966599 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.966741 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.966787 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:16:37.966856 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.964258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:43397 (local address 127.16.79.2:43997)
0620 23:16:37.964416 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:37.964425 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:37.964472 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:37.964613 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:37.964614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:37.964616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:37.964688 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.964690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.965344 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.965345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.966062 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.966064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.966592 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.966594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.966664 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.966668 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:16:37.966719 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 23:16:37.966746 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":5,"thread_start_us":104,"threads_started":1}
I20260620 23:16:37.966914 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.964222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:47919 (local address 127.16.79.1:40457)
0620 23:16:37.964331 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.964334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.964342 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:16:37.964372 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.964373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.964385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.964444 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:37.964633 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.965189 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.966158 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.966524 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.966610 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.966619 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.966805 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.966842 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:16:37.966929 2114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.964127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:40457 (local address 127.16.79.2:47919)
0620 23:16:37.964191 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:37.964203 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:37.964289 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:37.964476 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:37.964477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:37.964480 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:37.964555 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.964557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.965269 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.965271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.966064 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:37.966066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.966631 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.966632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:37.966721 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.966731 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:16:37.966787 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 23:16:37.966823 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":5}
I20260620 23:16:37.967440 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4ec58e199e4155a80d61f8699d6ec1" candidate_uuid: "19969717f9ab42a7bf40f70b9065561d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bb3b2c268b4d3495046015ea6685a8" is_pre_election: true
I20260620 23:16:37.967464 2198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4ec58e199e4155a80d61f8699d6ec1" candidate_uuid: "19969717f9ab42a7bf40f70b9065561d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed" is_pre_election: true
I20260620 23:16:37.967549 1935 raft_consensus.cc:2454] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19969717f9ab42a7bf40f70b9065561d in term 0.
I20260620 23:16:37.967621 2198 raft_consensus.cc:2454] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19969717f9ab42a7bf40f70b9065561d in term 0.
I20260620 23:16:37.967803 2001 leader_election.cc:302] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19969717f9ab42a7bf40f70b9065561d, 19bb3b2c268b4d3495046015ea6685a8; no voters:
I20260620 23:16:37.967914 2264 raft_consensus.cc:2781] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:16:37.967994 2264 raft_consensus.cc:482] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:16:37.968030 2264 raft_consensus.cc:3037] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:37.968791 2264 raft_consensus.cc:504] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19969717f9ab42a7bf40f70b9065561d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35737 } } peers { permanent_uuid: "19bb3b2c268b4d3495046015ea6685a8" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40457 } } peers { permanent_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43397 } }
I20260620 23:16:37.968924 2264 leader_election.cc:288] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [CANDIDATE]: Term 1 election: Requested vote from peers 19bb3b2c268b4d3495046015ea6685a8 (127.16.79.1:40457), 29d4ef3983f645b9a7f12ce7507dd3ed (127.16.79.3:43397)
I20260620 23:16:37.969257 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4ec58e199e4155a80d61f8699d6ec1" candidate_uuid: "19969717f9ab42a7bf40f70b9065561d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bb3b2c268b4d3495046015ea6685a8"
I20260620 23:16:37.969341 1935 raft_consensus.cc:3037] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:37.969394 2198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4ec58e199e4155a80d61f8699d6ec1" candidate_uuid: "19969717f9ab42a7bf40f70b9065561d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed"
I20260620 23:16:37.969525 2198 raft_consensus.cc:3037] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:37.969942 1935 raft_consensus.cc:2454] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19969717f9ab42a7bf40f70b9065561d in term 1.
I20260620 23:16:37.970146 2001 leader_election.cc:302] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19969717f9ab42a7bf40f70b9065561d, 19bb3b2c268b4d3495046015ea6685a8; no voters:
I20260620 23:16:37.970268 2264 raft_consensus.cc:2781] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:37.970388 2264 raft_consensus.cc:686] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 1 LEADER]: Becoming Leader. State: Replica: 19969717f9ab42a7bf40f70b9065561d, State: Running, Role: LEADER
I20260620 23:16:37.970449 2264 consensus_queue.cc:237] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19969717f9ab42a7bf40f70b9065561d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35737 } } peers { permanent_uuid: "19bb3b2c268b4d3495046015ea6685a8" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40457 } } peers { permanent_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43397 } }
I20260620 23:16:37.970741 2198 raft_consensus.cc:2454] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19969717f9ab42a7bf40f70b9065561d in term 1.
I20260620 23:16:37.971378 1793 catalog_manager.cc:5495] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d reported cstate change: term changed from 0 to 1, leader changed from <none> to 19969717f9ab42a7bf40f70b9065561d (127.16.79.2). New cstate: current_term: 1 leader_uuid: "19969717f9ab42a7bf40f70b9065561d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19969717f9ab42a7bf40f70b9065561d" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 35737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19bb3b2c268b4d3495046015ea6685a8" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43397 } health_report { overall_health: UNKNOWN } } }
I20260620 23:16:37.995605 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:37.993337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.16.79.2:35737)
0620 23:16:37.993397 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.993399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.993407 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:16:37.993458 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.993458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.993474 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.993533 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:37.993665 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.994129 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.994852 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:37.995038 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:37.995088 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:37.995236 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:37.995322 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:37.995388 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:37.995470 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 23:16:37.995485 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:16:37.997871 1935 raft_consensus.cc:1261] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19bb3b2c268b4d3495046015ea6685a8 [term 1 FOLLOWER]: Refusing update from remote peer 19969717f9ab42a7bf40f70b9065561d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:37.997871 2198 raft_consensus.cc:1261] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed [term 1 FOLLOWER]: Refusing update from remote peer 19969717f9ab42a7bf40f70b9065561d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:37.998240 2268 consensus_queue.cc:1035] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19bb3b2c268b4d3495046015ea6685a8" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 40457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:37.998378 2264 consensus_queue.cc:1035] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d4ef3983f645b9a7f12ce7507dd3ed" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:38.001013 2272 mvcc.cc:205] Tried to move back new op lower bound from 7299061342196568064 to 7299061342087479296. Current Snapshot: MvccSnapshot[applied={T|T < 7299061342196568064}]
I20260620 23:16:38.001662 2273 mvcc.cc:205] Tried to move back new op lower bound from 7299061342196568064 to 7299061342087479296. Current Snapshot: MvccSnapshot[applied={T|T < 7299061342196568064}]
I20260620 23:16:38.002599 1793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:16:38.002700 1793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:16:38.003409 1793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff98bf4fdc5048629568e2e0c6ef946b: Sending DeleteTablet for 3 replicas of tablet 4a4ec58e199e4155a80d61f8699d6ec1
I20260620 23:16:38.003769 1915 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a4ec58e199e4155a80d61f8699d6ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:38 UTC) from {username='slave'} at 127.0.0.1:53828
I20260620 23:16:38.003798 2046 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a4ec58e199e4155a80d61f8699d6ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:38 UTC) from {username='slave'} at 127.0.0.1:33318
I20260620 23:16:38.003839 2178 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a4ec58e199e4155a80d61f8699d6ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:38 UTC) from {username='slave'} at 127.0.0.1:35616
I20260620 23:16:38.004169 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 1853
I20260620 23:16:38.012668 2283 tablet_replica.cc:330] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed: stopping tablet replica
I20260620 23:16:38.012684 2281 tablet_replica.cc:330] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d: stopping tablet replica
I20260620 23:16:38.012800 2283 raft_consensus.cc:2229] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:16:38.012822 2281 raft_consensus.cc:2229] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:16:38.012903 2283 raft_consensus.cc:2258] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:38.012912 2281 raft_consensus.cc:2258] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 23:16:38.013137 1779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:40457 (error 108)
I20260620 23:16:38.013376 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 1984
I20260620 23:16:38.013376 2281 ts_tablet_manager.cc:1905] T 4a4ec58e199e4155a80d61f8699d6ec1 P 19969717f9ab42a7bf40f70b9065561d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:16:38.014396 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:38.014256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:40457 (local address 127.0.0.1:53840)
0620 23:16:38.014313 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:38.014357 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:40457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:16:38.014521 2283 ts_tablet_manager.cc:1905] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 23:16:38.014591 1779 catalog_manager.cc:4584] TS 19bb3b2c268b4d3495046015ea6685a8 (127.16.79.1:40457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a4ec58e199e4155a80d61f8699d6ec1: Network error: Client connection negotiation failed: client connection to 127.16.79.1:40457: connect: Connection refused (error 111)
I20260620 23:16:38.016145 2283 ts_tablet_manager.cc:1918] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:16:38.016232 2283 log.cc:1196] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/4a4ec58e199e4155a80d61f8699d6ec1
I20260620 23:16:38.016464 2283 ts_tablet_manager.cc:1939] T 4a4ec58e199e4155a80d61f8699d6ec1 P 29d4ef3983f645b9a7f12ce7507dd3ed: Deleting consensus metadata
I20260620 23:16:38.016930 1779 catalog_manager.cc:4841] TS 29d4ef3983f645b9a7f12ce7507dd3ed (127.16.79.3:43397): tablet 4a4ec58e199e4155a80d61f8699d6ec1 (table test-table [id=6855a7c4e5e14ec6be3752c68e8ffb2c]) successfully deleted
I20260620 23:16:38.018661 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 2115
I20260620 23:16:38.018731 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:38.018601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:35737 (local address 127.0.0.1:33350)
0620 23:16:38.018654 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:38.018686 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.2:35737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 23:16:38.022972 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 1762
2026-06-20T23:16:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (964 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:38 dist-test-slave-pswb krb5kdc[2287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:38 dist-test-slave-pswb krb5kdc[2287](info): set up 2 sockets
Jun 20 23:16:38 dist-test-slave-pswb krb5kdc[2287](info): commencing operation
krb5kdc: starting...
W20260620 23:16:39.080889 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:39 dist-test-slave-pswb krb5kdc[2287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:16:39.204577 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:38563
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:38563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:39.284125 2300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:39.284351 2300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:39.284387 2300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:39.285527 2300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:39.285596 2300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:39.285622 2300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:39.285655 2300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:39.285678 2300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:39.287077 2300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:38563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:38563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:39.287339 2300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:39.287552 2300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:39.292524 2308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:39.292559 2306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:39.292578 2305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:39.292908 2300 server_base.cc:1034] running on GCE node
I20260620 23:16:39.293139 2300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:39.293406 2300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:39.294566 2300 hybrid_clock.cc:648] HybridClock initialized: now 1781997399294598 us; error 111 us; skew 500 ppm
Jun 20 23:16:39 dist-test-slave-pswb krb5kdc[2287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:39.303462 2300 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:16:39.303915 2300 webserver.cc:460] Webserver started at http://127.16.79.62:39521/ using document root <none> and password file <none>
I20260620 23:16:39.304157 2300 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:39.304227 2300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:39.304378 2300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:39.305456 2300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "49f9f3099fba44a3b88229b39a77e7be"
format_stamp: "Formatted at 2026-06-20 23:16:39 on dist-test-slave-pswb"
server_key: "d78fe181d6692c0d3e5ad31f9c9e9efb"
server_key_iv: "c7de517dd245c4ba8bd16b5f098a10dd"
server_key_version: "encryptionkey@0"
I20260620 23:16:39.305817 2300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "49f9f3099fba44a3b88229b39a77e7be"
format_stamp: "Formatted at 2026-06-20 23:16:39 on dist-test-slave-pswb"
server_key: "d78fe181d6692c0d3e5ad31f9c9e9efb"
server_key_iv: "c7de517dd245c4ba8bd16b5f098a10dd"
server_key_version: "encryptionkey@0"
I20260620 23:16:39.307240 2300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:16:39.308463 2315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:39.308853 2300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:39.309036 2300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "49f9f3099fba44a3b88229b39a77e7be"
format_stamp: "Formatted at 2026-06-20 23:16:39 on dist-test-slave-pswb"
server_key: "d78fe181d6692c0d3e5ad31f9c9e9efb"
server_key_iv: "c7de517dd245c4ba8bd16b5f098a10dd"
server_key_version: "encryptionkey@0"
I20260620 23:16:39.309190 2300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:39.322710 2300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:39.324887 2300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:39.325050 2300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:39.329532 2300 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:38563
I20260620 23:16:39.329563 2367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:38563 every 8 connection(s)
I20260620 23:16:39.329926 2300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:39.330080 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 2300
I20260620 23:16:39.330193 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:39.330405 16700 external_mini_cluster.cc:1428] Setting key fda5cbabfc4306271470f935b6b4b4d1
I20260620 23:16:39.331050 2368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:39.334872 2368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be: Bootstrap starting.
I20260620 23:16:39.335987 2368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:39.336365 2368 log.cc:824] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:39.337278 2368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be: No bootstrap required, opened a new log
Jun 20 23:16:39 dist-test-slave-pswb krb5kdc[2287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:16:39.339624 2368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f9f3099fba44a3b88229b39a77e7be" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 38563 } }
I20260620 23:16:39.339788 2368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:39.339840 2368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f9f3099fba44a3b88229b39a77e7be, State: Initialized, Role: FOLLOWER
I20260620 23:16:39.339994 2368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f9f3099fba44a3b88229b39a77e7be" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 38563 } }
I20260620 23:16:39.340083 2368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:39.340126 2368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:39.340178 2368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:39.340795 2368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f9f3099fba44a3b88229b39a77e7be" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 38563 } }
I20260620 23:16:39.340917 2368 leader_election.cc:302] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f9f3099fba44a3b88229b39a77e7be; no voters:
I20260620 23:16:39.341089 2368 leader_election.cc:288] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:39.341128 2373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:39.341348 2373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [term 1 LEADER]: Becoming Leader. State: Replica: 49f9f3099fba44a3b88229b39a77e7be, State: Running, Role: LEADER
I20260620 23:16:39.341449 2373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f9f3099fba44a3b88229b39a77e7be" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 38563 } }
I20260620 23:16:39.341355 2368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:39.341746 2373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f9f3099fba44a3b88229b39a77e7be. Latest consensus state: current_term: 1 leader_uuid: "49f9f3099fba44a3b88229b39a77e7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f9f3099fba44a3b88229b39a77e7be" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 38563 } } }
I20260620 23:16:39.341753 2375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f9f3099fba44a3b88229b39a77e7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f9f3099fba44a3b88229b39a77e7be" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 38563 } } }
I20260620 23:16:39.341876 2373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:39.341952 2375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:39.342187 2378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:39.342733 2378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:39.343083 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.330907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.16.79.62:38563)
0620 23:16:39.331111 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:39.331121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:39.331154 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:16:39.331779 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:39.331788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:39.331851 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:39.332013 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:39.332672 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.334365 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.335318 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.336199 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.336353 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.339026 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:39.339030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:39.339030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:39.339040 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:39.340971 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:39.341400 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:39.341403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:39.341403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:39.341502 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:39.341734 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:39.341734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:39.341735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:39.342031 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:39.342604 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:39.342695 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 23:16:39.342846 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20260620 23:16:39.345220 2378 catalog_manager.cc:1347] Generated new cluster ID: a710611114cc43a1843cf6590b0bcdcc
I20260620 23:16:39.345280 2378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:39.351512 2378 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:39.352190 2378 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:39.360913 2378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be: Generated new TSK 0
I20260620 23:16:39.361096 2378 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:16:39.388590 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:38563
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:39.475432 2394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:39.475628 2394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:39.475652 2394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:39.476825 2394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:39.476866 2394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:39.476893 2394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:16:39.478319 2394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:39.478549 2394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:39.478799 2394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:39.483680 2400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:39.483827 2399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:39.483893 2402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:39.483982 2394 server_base.cc:1034] running on GCE node
I20260620 23:16:39.484239 2394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:39.484488 2394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:39.485659 2394 hybrid_clock.cc:648] HybridClock initialized: now 1781997399485637 us; error 64 us; skew 500 ppm
Jun 20 23:16:39 dist-test-slave-pswb krb5kdc[2287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:39.489014 2394 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:16:39.489631 2394 webserver.cc:460] Webserver started at http://127.16.79.1:46149/ using document root <none> and password file <none>
I20260620 23:16:39.489890 2394 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:39.490101 2394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:39.490270 2394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:39.491282 2394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "04b3253a7c8b4e4e9382897577bc1cef"
format_stamp: "Formatted at 2026-06-20 23:16:39 on dist-test-slave-pswb"
server_key: "06ac578a4598e9e9f90778d4b6d93bbe"
server_key_iv: "d663aa1ba829202f71cb5f310f7fc1f1"
server_key_version: "encryptionkey@0"
I20260620 23:16:39.491638 2394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "04b3253a7c8b4e4e9382897577bc1cef"
format_stamp: "Formatted at 2026-06-20 23:16:39 on dist-test-slave-pswb"
server_key: "06ac578a4598e9e9f90778d4b6d93bbe"
server_key_iv: "d663aa1ba829202f71cb5f310f7fc1f1"
server_key_version: "encryptionkey@0"
I20260620 23:16:39.493155 2394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:16:39.494367 2409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:39.494585 2394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:39.494689 2394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "04b3253a7c8b4e4e9382897577bc1cef"
format_stamp: "Formatted at 2026-06-20 23:16:39 on dist-test-slave-pswb"
server_key: "06ac578a4598e9e9f90778d4b6d93bbe"
server_key_iv: "d663aa1ba829202f71cb5f310f7fc1f1"
server_key_version: "encryptionkey@0"
I20260620 23:16:39.494776 2394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:39.506647 2394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:39.508906 2394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:39.509064 2394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:39.509286 2394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:39.509580 2394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:39.509637 2394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:39.509675 2394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:39.509703 2394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:39.515838 2394 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:36631
I20260620 23:16:39.515884 2522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:36631 every 8 connection(s)
I20260620 23:16:39.516172 2394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:16:39 dist-test-slave-pswb krb5kdc[2287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:16:39.524250 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 2394
I20260620 23:16:39.524380 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:16:39.524585 16700 external_mini_cluster.cc:1428] Setting key 2c867da06fb2c3c3d32d52fe9cf31194
I20260620 23:16:39.526247 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.516812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:38563 (local address 127.16.79.1:51625)
0620 23:16:39.516975 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:39.517008 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:39.517492 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:39.517741 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:39.517744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:39.518297 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:39.518474 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:39.518478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.519311 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.519312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:39.520102 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:39.520104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.520511 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.520512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:39.520635 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.521189 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:39.521198 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:39.522704 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:39.524934 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:39.524941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:39.524942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:39.525157 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:39.525381 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:39.525382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:39.525382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:39.525485 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:39.525786 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:39.525791 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:39.525901 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 23:16:39.525985 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:16:39.526098 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260620 23:16:39.526258 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.516831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:51625 (local address 127.16.79.62:38563)
0620 23:16:39.516873 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:39.516875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:39.517092 (+ 217us) server_negotiation.cc:407] Connection header received
0620 23:16:39.517606 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:39.517607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:39.517620 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:39.517676 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:39.518574 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.519216 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.520201 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.520420 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.520491 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.522879 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:39.522881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:39.522882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:39.522893 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:39.524812 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:39.525240 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:39.525241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:39.525242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:39.525310 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:39.525562 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:39.525563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:39.525563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:39.525715 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:39.525786 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:39.526029 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 23:16:39.526119 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:16:39.526950 2523 heartbeater.cc:347] Connected to a master server at 127.16.79.62:38563
I20260620 23:16:39.527033 2523 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:39.527210 2523 heartbeater.cc:511] Master 127.16.79.62:38563 requested a full tablet report, sending...
I20260620 23:16:39.527740 2331 ts_manager.cc:194] Registered new tserver with Master: 04b3253a7c8b4e4e9382897577bc1cef (127.16.79.1:36631)
I20260620 23:16:39.528296 2331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:51625
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:16:39.552547 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:38563
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:39.637091 2528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:39.637281 2528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:39.637306 2528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:39.638406 2528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:39.638443 2528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:39.638469 2528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:16:39.639814 2528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:39.639989 2528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:39.640146 2528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:16:39.645283 2528 server_base.cc:1034] running on GCE node
W20260620 23:16:39.645427 2534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:39.645344 2536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:39.645280 2533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:39.645790 2528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:39.646045 2528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:39.647228 2528 hybrid_clock.cc:648] HybridClock initialized: now 1781997399647169 us; error 61 us; skew 500 ppm
Jun 20 23:16:39 dist-test-slave-pswb krb5kdc[2287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:39.650238 2528 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:16:39.650794 2528 webserver.cc:460] Webserver started at http://127.16.79.2:41827/ using document root <none> and password file <none>
I20260620 23:16:39.651077 2528 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:39.651131 2528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:39.651279 2528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:39.652536 2528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9"
format_stamp: "Formatted at 2026-06-20 23:16:39 on dist-test-slave-pswb"
server_key: "dcb6ca2c880e7ae0566594a2fa6ec4dc"
server_key_iv: "fd6315bc4c687c1037afdb855bcabf4a"
server_key_version: "encryptionkey@0"
I20260620 23:16:39.652977 2528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9"
format_stamp: "Formatted at 2026-06-20 23:16:39 on dist-test-slave-pswb"
server_key: "dcb6ca2c880e7ae0566594a2fa6ec4dc"
server_key_iv: "fd6315bc4c687c1037afdb855bcabf4a"
server_key_version: "encryptionkey@0"
I20260620 23:16:39.654312 2528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:16:39.655056 2543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:39.655233 2528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:39.655319 2528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9"
format_stamp: "Formatted at 2026-06-20 23:16:39 on dist-test-slave-pswb"
server_key: "dcb6ca2c880e7ae0566594a2fa6ec4dc"
server_key_iv: "fd6315bc4c687c1037afdb855bcabf4a"
server_key_version: "encryptionkey@0"
I20260620 23:16:39.655371 2528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:39.681242 2528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:39.683463 2528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:39.683573 2528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:39.683743 2528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:39.684067 2528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:39.684108 2528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:39.684129 2528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:39.684141 2528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:39.690142 2528 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:43739
I20260620 23:16:39.690228 2656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:43739 every 8 connection(s)
I20260620 23:16:39.690539 2528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:16:39 dist-test-slave-pswb krb5kdc[2287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:16:39.698238 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 2528
I20260620 23:16:39.698355 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:16:39.698624 16700 external_mini_cluster.cc:1428] Setting key f69ce006a22450ca7c4fbe88d044eef6
I20260620 23:16:39.699992 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.691343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:48223 (local address 127.16.79.62:38563)
0620 23:16:39.691421 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:39.691424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:39.691589 (+ 165us) server_negotiation.cc:407] Connection header received
0620 23:16:39.692099 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:39.692100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:39.692114 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:39.692159 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:39.693152 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.693649 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.694613 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.694808 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.694884 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.696991 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:39.696993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:39.696993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:39.696999 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:39.698537 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:39.699023 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:39.699025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:39.699025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:39.699094 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:39.699385 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:39.699385 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:39.699386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:39.699490 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:39.699544 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:39.699870 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 23:16:39.699931 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:16:39.700320 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.691289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:38563 (local address 127.16.79.2:48223)
0620 23:16:39.691475 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:39.691509 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:39.691987 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:39.692238 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:39.692241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:39.692870 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:39.693049 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:39.693054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.693742 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.693743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:39.694508 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:39.694510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.694892 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.694893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:39.695011 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.695435 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:39.695442 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:39.696873 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:39.698621 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:39.698631 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:39.698633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:39.698906 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:39.699163 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:39.699164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:39.699164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:39.699313 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:39.699556 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:39.699562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:39.699709 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 23:16:39.699837 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 23:16:39.700009 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260620 23:16:39.700827 2657 heartbeater.cc:347] Connected to a master server at 127.16.79.62:38563
I20260620 23:16:39.700933 2657 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:39.701095 2657 heartbeater.cc:511] Master 127.16.79.62:38563 requested a full tablet report, sending...
I20260620 23:16:39.701474 2331 ts_manager.cc:194] Registered new tserver with Master: 16a2c5fe9fcd4ccaa31008f35fced6f9 (127.16.79.2:43739)
I20260620 23:16:39.701938 2331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:48223
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:16:39.728470 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:38563
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:39.810127 2662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:39.810391 2662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:39.810506 2662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:39.813352 2662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:39.813416 2662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:39.813460 2662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:16:39.815244 2662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:39.815500 2662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:39.815738 2662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:39.820849 2667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:39.821059 2668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:39.821228 2670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:39.821782 2662 server_base.cc:1034] running on GCE node
I20260620 23:16:39.822042 2662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:39.822342 2662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:39.823575 2662 hybrid_clock.cc:648] HybridClock initialized: now 1781997399823546 us; error 40 us; skew 500 ppm
Jun 20 23:16:39 dist-test-slave-pswb krb5kdc[2287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:39.827082 2662 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:16:39.827555 2662 webserver.cc:460] Webserver started at http://127.16.79.3:40049/ using document root <none> and password file <none>
I20260620 23:16:39.827759 2662 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:39.827801 2662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:39.827893 2662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:39.829068 2662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "13976fdd9a11464da3e3d775e968e9bf"
format_stamp: "Formatted at 2026-06-20 23:16:39 on dist-test-slave-pswb"
server_key: "aca1094ff5c6bf5d356cea77b393d9c6"
server_key_iv: "fe07b860682a0a3de9e31cd949ad5b3b"
server_key_version: "encryptionkey@0"
I20260620 23:16:39.829368 2662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "13976fdd9a11464da3e3d775e968e9bf"
format_stamp: "Formatted at 2026-06-20 23:16:39 on dist-test-slave-pswb"
server_key: "aca1094ff5c6bf5d356cea77b393d9c6"
server_key_iv: "fe07b860682a0a3de9e31cd949ad5b3b"
server_key_version: "encryptionkey@0"
I20260620 23:16:39.830667 2662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:16:39.831375 2677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:39.831580 2662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:39.831672 2662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "13976fdd9a11464da3e3d775e968e9bf"
format_stamp: "Formatted at 2026-06-20 23:16:39 on dist-test-slave-pswb"
server_key: "aca1094ff5c6bf5d356cea77b393d9c6"
server_key_iv: "fe07b860682a0a3de9e31cd949ad5b3b"
server_key_version: "encryptionkey@0"
I20260620 23:16:39.831738 2662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:39.856557 2662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:39.858913 2662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:39.859023 2662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:39.859252 2662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:39.859586 2662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:39.859617 2662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:39.859640 2662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:39.859689 2662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:39.865564 2662 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:35233
I20260620 23:16:39.865607 2791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:35233 every 8 connection(s)
I20260620 23:16:39.865902 2662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:16:39 dist-test-slave-pswb krb5kdc[2287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:16:39.873469 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 2662
I20260620 23:16:39.873581 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:16:39.873776 16700 external_mini_cluster.cc:1428] Setting key 868b2365dfec95771f46c05d99b9f3ec
I20260620 23:16:39.876427 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.866678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:38697 (local address 127.16.79.62:38563)
0620 23:16:39.866749 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:39.866752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:39.866902 (+ 150us) server_negotiation.cc:407] Connection header received
0620 23:16:39.867721 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:39.867723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:39.867745 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:39.867791 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:39.869145 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.869658 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.870667 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.870863 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.870936 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.873329 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:39.873330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:39.873331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:39.873338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:39.874977 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:39.875468 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:39.875469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:39.875470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:39.875529 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:39.875787 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:39.875788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:39.875788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:39.875918 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:39.875998 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:39.876263 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 23:16:39.876339 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:16:39.876612 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.866616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:38563 (local address 127.16.79.3:38697)
0620 23:16:39.866771 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:39.866815 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:39.867322 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:39.867892 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:39.867904 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:39.868823 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:39.869023 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:39.869029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.869761 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.869762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:39.870537 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:39.870539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.870953 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.870954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:39.871074 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.871575 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:39.871583 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:39.873228 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:39.875067 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:39.875128 (+ 61us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:39.875138 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:39.875359 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:39.875606 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:39.875607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:39.875607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:39.875717 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:39.875988 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:39.875994 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:39.876130 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 23:16:39.876230 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 23:16:39.876390 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260620 23:16:39.877211 2792 heartbeater.cc:347] Connected to a master server at 127.16.79.62:38563
I20260620 23:16:39.877420 2792 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:39.877770 2792 heartbeater.cc:511] Master 127.16.79.62:38563 requested a full tablet report, sending...
I20260620 23:16:39.878278 2331 ts_manager.cc:194] Registered new tserver with Master: 13976fdd9a11464da3e3d775e968e9bf (127.16.79.3:35233)
I20260620 23:16:39.878604 2331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:38697
I20260620 23:16:39.885864 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:16:39.895367 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.887758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.16.79.62:38563)
0620 23:16:39.887835 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:39.887837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:39.887849 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:16:39.887872 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:39.887873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:39.887891 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:39.887957 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:39.888785 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.889344 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.890318 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.890651 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.890716 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.892155 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:39.892156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:39.892156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:39.892165 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:39.893966 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:39.894374 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:39.894374 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:39.894375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:39.894428 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:39.894675 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:39.894676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:39.894677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:39.894879 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:39.895009 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:39.895135 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 23:16:39.895204 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:16:39.897322 2331 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:16:39.899000 2331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:16:39.906095 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.901466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:35233 (local address 127.0.0.1:56094)
0620 23:16:39.901817 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:39.901859 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:39.901930 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:39.902147 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:39.902151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:39.902155 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:39.902269 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:39.902271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.903566 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.903568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:39.904771 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:39.904774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.905729 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.905730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:39.905813 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.905829 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 23:16:39.905922 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:16:39.905993 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":95,"threads_started":1}
I20260620 23:16:39.906101 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.901641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.16.79.3:35233)
0620 23:16:39.901905 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:39.901911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:39.901923 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:16:39.902019 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:39.902022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:39.902039 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:39.902123 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:39.902375 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.903437 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.904864 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.905641 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.905716 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.905781 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:39.905944 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:16:39.905996 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":128,"threads_started":1}
I20260620 23:16:39.906318 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.901528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:36631 (local address 127.0.0.1:58900)
0620 23:16:39.902195 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:39.902205 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:39.902247 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:39.902730 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:39.902732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:39.902735 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:39.902784 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:39.902786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.904134 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.904135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:39.905212 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:39.905214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.906080 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.906081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:39.906144 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.906149 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:16:39.906198 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 23:16:39.906242 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":85,"spinlock_wait_cycles":768,"thread_start_us":114,"threads_started":1}
I20260620 23:16:39.906631 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.902118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.16.79.1:36631)
0620 23:16:39.902322 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:39.902327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:39.902337 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:16:39.902376 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:39.902378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:39.902393 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:39.902626 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:39.902862 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.904013 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.905301 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.905932 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.905994 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.906047 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:39.906492 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 23:16:39.906546 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
I20260620 23:16:39.907011 2725 tablet_service.cc:1467] Processing CreateTablet for tablet c57f42058228469c8922b522750eca4b (DEFAULT_TABLE table=test-table [id=f9ae7e19e08b4fac974c89870bcc0046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:39.907212 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.901528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:43739 (local address 127.0.0.1:37244)
0620 23:16:39.902040 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:39.902048 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:39.902103 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:39.902651 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:39.902653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:39.902655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:39.902706 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:39.902707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.904142 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.904143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:39.905409 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:39.905413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.906850 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.906851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:39.907042 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.907047 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:16:39.907102 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 23:16:39.907143 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":35,"thread_start_us":111,"threads_started":1}
I20260620 23:16:39.907207 2457 tablet_service.cc:1467] Processing CreateTablet for tablet c57f42058228469c8922b522750eca4b (DEFAULT_TABLE table=test-table [id=f9ae7e19e08b4fac974c89870bcc0046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:39.907405 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:39.901845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.16.79.2:43739)
0620 23:16:39.902045 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:39.902051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:39.902377 (+ 326us) server_negotiation.cc:407] Connection header received
0620 23:16:39.902428 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:39.902431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:39.902449 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:39.902574 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:39.902808 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.904004 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.905512 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:39.906688 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:39.906759 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:39.906816 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:39.907123 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 23:16:39.907169 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":117,"threads_started":1}
I20260620 23:16:39.907459 2725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57f42058228469c8922b522750eca4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:39.907505 2457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57f42058228469c8922b522750eca4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:39.908078 2591 tablet_service.cc:1467] Processing CreateTablet for tablet c57f42058228469c8922b522750eca4b (DEFAULT_TABLE table=test-table [id=f9ae7e19e08b4fac974c89870bcc0046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:39.908387 2591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57f42058228469c8922b522750eca4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:39.911448 2809 tablet_bootstrap.cc:492] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf: Bootstrap starting.
I20260620 23:16:39.911826 2811 tablet_bootstrap.cc:492] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9: Bootstrap starting.
I20260620 23:16:39.911873 2810 tablet_bootstrap.cc:492] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef: Bootstrap starting.
I20260620 23:16:39.912487 2811 tablet_bootstrap.cc:654] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:39.912501 2810 tablet_bootstrap.cc:654] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:39.913149 2811 log.cc:824] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:39.913162 2810 log.cc:824] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:39.913489 2809 tablet_bootstrap.cc:654] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:39.913915 2811 tablet_bootstrap.cc:492] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9: No bootstrap required, opened a new log
I20260620 23:16:39.914003 2811 ts_tablet_manager.cc:1397] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:16:39.914085 2809 log.cc:824] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:39.914155 2810 tablet_bootstrap.cc:492] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef: No bootstrap required, opened a new log
I20260620 23:16:39.914217 2810 ts_tablet_manager.cc:1397] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:16:39.915544 2810 raft_consensus.cc:348] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13976fdd9a11464da3e3d775e968e9bf" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 35233 } } peers { permanent_uuid: "04b3253a7c8b4e4e9382897577bc1cef" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36631 } } peers { permanent_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43739 } }
I20260620 23:16:39.915683 2810 raft_consensus.cc:374] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:39.915565 2811 raft_consensus.cc:348] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13976fdd9a11464da3e3d775e968e9bf" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 35233 } } peers { permanent_uuid: "04b3253a7c8b4e4e9382897577bc1cef" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36631 } } peers { permanent_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43739 } }
I20260620 23:16:39.915705 2810 raft_consensus.cc:729] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b3253a7c8b4e4e9382897577bc1cef, State: Initialized, Role: FOLLOWER
I20260620 23:16:39.915714 2811 raft_consensus.cc:374] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:39.915751 2811 raft_consensus.cc:729] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a2c5fe9fcd4ccaa31008f35fced6f9, State: Initialized, Role: FOLLOWER
I20260620 23:16:39.915864 2811 consensus_queue.cc:260] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13976fdd9a11464da3e3d775e968e9bf" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 35233 } } peers { permanent_uuid: "04b3253a7c8b4e4e9382897577bc1cef" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36631 } } peers { permanent_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43739 } }
I20260620 23:16:39.915864 2810 consensus_queue.cc:260] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13976fdd9a11464da3e3d775e968e9bf" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 35233 } } peers { permanent_uuid: "04b3253a7c8b4e4e9382897577bc1cef" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36631 } } peers { permanent_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43739 } }
I20260620 23:16:39.916126 2810 ts_tablet_manager.cc:1428] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:16:39.916127 2811 ts_tablet_manager.cc:1428] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:16:39.916145 2523 heartbeater.cc:503] Master 127.16.79.62:38563 was elected leader, sending a full tablet report...
I20260620 23:16:39.916272 2657 heartbeater.cc:503] Master 127.16.79.62:38563 was elected leader, sending a full tablet report...
I20260620 23:16:39.916767 2809 tablet_bootstrap.cc:492] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf: No bootstrap required, opened a new log
I20260620 23:16:39.916846 2809 ts_tablet_manager.cc:1397] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 23:16:39.918581 2809 raft_consensus.cc:348] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13976fdd9a11464da3e3d775e968e9bf" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 35233 } } peers { permanent_uuid: "04b3253a7c8b4e4e9382897577bc1cef" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36631 } } peers { permanent_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43739 } }
I20260620 23:16:39.918709 2809 raft_consensus.cc:374] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:39.918730 2809 raft_consensus.cc:729] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13976fdd9a11464da3e3d775e968e9bf, State: Initialized, Role: FOLLOWER
I20260620 23:16:39.918844 2809 consensus_queue.cc:260] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13976fdd9a11464da3e3d775e968e9bf" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 35233 } } peers { permanent_uuid: "04b3253a7c8b4e4e9382897577bc1cef" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36631 } } peers { permanent_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43739 } }
I20260620 23:16:39.919064 2809 ts_tablet_manager.cc:1428] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:16:39.919148 2792 heartbeater.cc:503] Master 127.16.79.62:38563 was elected leader, sending a full tablet report...
W20260620 23:16:39.941097 2658 tablet.cc:2369] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:16:40.016923 2524 tablet.cc:2369] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:16:40.077755 2817 raft_consensus.cc:482] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:16:40.078158 2817 raft_consensus.cc:504] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13976fdd9a11464da3e3d775e968e9bf" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 35233 } } peers { permanent_uuid: "04b3253a7c8b4e4e9382897577bc1cef" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36631 } } peers { permanent_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43739 } }
I20260620 23:16:40.078831 2817 leader_election.cc:288] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04b3253a7c8b4e4e9382897577bc1cef (127.16.79.1:36631), 16a2c5fe9fcd4ccaa31008f35fced6f9 (127.16.79.2:43739)
I20260620 23:16:40.082060 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.079224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:58601 (local address 127.16.79.2:43739)
0620 23:16:40.079307 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.079309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.079408 (+ 99us) server_negotiation.cc:407] Connection header received
0620 23:16:40.079480 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.079481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.079501 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.079567 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:40.079751 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.080258 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.081336 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.081714 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.081798 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.081809 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.081960 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.081997 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:16:40.082077 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.079090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:43739 (local address 127.16.79.3:58601)
0620 23:16:40.079330 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.079339 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.079422 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.079569 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.079570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.079572 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:40.079670 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.079672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.080353 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.080354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.081235 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.081238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.081800 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.081801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.081876 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.081880 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.081943 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.081986 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":24,"thread_start_us":123,"threads_started":1}
I20260620 23:16:40.082393 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.079090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:36631 (local address 127.16.79.3:49403)
0620 23:16:40.079153 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.079167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.079216 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.079656 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.079658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.079661 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:40.079765 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.079767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.080804 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.080805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.081536 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.081538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.082131 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.082132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.082214 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.082217 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.082271 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.082327 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260620 23:16:40.082464 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.079225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:49403 (local address 127.16.79.1:36631)
0620 23:16:40.079278 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.079280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.079292 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:16:40.079401 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.079403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.079435 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.079640 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:40.079854 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.080666 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.081641 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.082033 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.082112 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.082123 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.082348 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.082389 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:16:40.082767 2611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57f42058228469c8922b522750eca4b" candidate_uuid: "13976fdd9a11464da3e3d775e968e9bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9" is_pre_election: true
I20260620 23:16:40.082947 2611 raft_consensus.cc:2454] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13976fdd9a11464da3e3d775e968e9bf in term 0.
I20260620 23:16:40.082894 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57f42058228469c8922b522750eca4b" candidate_uuid: "13976fdd9a11464da3e3d775e968e9bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b3253a7c8b4e4e9382897577bc1cef" is_pre_election: true
I20260620 23:16:40.083175 2477 raft_consensus.cc:2454] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13976fdd9a11464da3e3d775e968e9bf in term 0.
I20260620 23:16:40.083312 2680 leader_election.cc:302] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13976fdd9a11464da3e3d775e968e9bf, 16a2c5fe9fcd4ccaa31008f35fced6f9; no voters:
I20260620 23:16:40.083521 2817 raft_consensus.cc:2781] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:16:40.083611 2817 raft_consensus.cc:482] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:16:40.083639 2817 raft_consensus.cc:3037] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:40.084345 2817 raft_consensus.cc:504] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13976fdd9a11464da3e3d775e968e9bf" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 35233 } } peers { permanent_uuid: "04b3253a7c8b4e4e9382897577bc1cef" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36631 } } peers { permanent_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43739 } }
I20260620 23:16:40.084503 2817 leader_election.cc:288] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [CANDIDATE]: Term 1 election: Requested vote from peers 04b3253a7c8b4e4e9382897577bc1cef (127.16.79.1:36631), 16a2c5fe9fcd4ccaa31008f35fced6f9 (127.16.79.2:43739)
I20260620 23:16:40.084713 2611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57f42058228469c8922b522750eca4b" candidate_uuid: "13976fdd9a11464da3e3d775e968e9bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9"
I20260620 23:16:40.084767 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57f42058228469c8922b522750eca4b" candidate_uuid: "13976fdd9a11464da3e3d775e968e9bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b3253a7c8b4e4e9382897577bc1cef"
I20260620 23:16:40.084833 2611 raft_consensus.cc:3037] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:40.084846 2477 raft_consensus.cc:3037] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:40.085592 2477 raft_consensus.cc:2454] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13976fdd9a11464da3e3d775e968e9bf in term 1.
I20260620 23:16:40.085695 2611 raft_consensus.cc:2454] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13976fdd9a11464da3e3d775e968e9bf in term 1.
I20260620 23:16:40.085773 2678 leader_election.cc:302] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04b3253a7c8b4e4e9382897577bc1cef, 13976fdd9a11464da3e3d775e968e9bf; no voters:
I20260620 23:16:40.085853 2817 raft_consensus.cc:2781] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:40.086006 2817 raft_consensus.cc:686] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 1 LEADER]: Becoming Leader. State: Replica: 13976fdd9a11464da3e3d775e968e9bf, State: Running, Role: LEADER
I20260620 23:16:40.086134 2817 consensus_queue.cc:237] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13976fdd9a11464da3e3d775e968e9bf" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 35233 } } peers { permanent_uuid: "04b3253a7c8b4e4e9382897577bc1cef" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36631 } } peers { permanent_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43739 } }
I20260620 23:16:40.087096 2332 catalog_manager.cc:5495] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 13976fdd9a11464da3e3d775e968e9bf (127.16.79.3). New cstate: current_term: 1 leader_uuid: "13976fdd9a11464da3e3d775e968e9bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13976fdd9a11464da3e3d775e968e9bf" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04b3253a7c8b4e4e9382897577bc1cef" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43739 } health_report { overall_health: UNKNOWN } } }
W20260620 23:16:40.116425 2793 tablet.cc:2369] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:16:40.134266 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.131394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.16.79.3:35233)
0620 23:16:40.131444 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.131446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.131456 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:16:40.131501 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.131502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.131520 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.131601 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:40.131731 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.132194 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.133315 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.133551 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.133658 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.133843 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:40.133973 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:40.134095 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.134126 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.134136 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:16:40.136224 2611 raft_consensus.cc:1261] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9 [term 1 FOLLOWER]: Refusing update from remote peer 13976fdd9a11464da3e3d775e968e9bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:40.136227 2477 raft_consensus.cc:1261] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef [term 1 FOLLOWER]: Refusing update from remote peer 13976fdd9a11464da3e3d775e968e9bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:40.136618 2817 consensus_queue.cc:1035] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a2c5fe9fcd4ccaa31008f35fced6f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:40.136812 2819 consensus_queue.cc:1035] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04b3253a7c8b4e4e9382897577bc1cef" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:40.139981 2823 mvcc.cc:205] Tried to move back new op lower bound from 7299061350954954752 to 7299061350753546240. Current Snapshot: MvccSnapshot[applied={T|T < 7299061350954954752 or (T in {7299061350954954752})}]
I20260620 23:16:40.141314 2331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:16:40.141424 2331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:16:40.142122 2331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49f9f3099fba44a3b88229b39a77e7be: Sending DeleteTablet for 3 replicas of tablet c57f42058228469c8922b522750eca4b
I20260620 23:16:40.142455 2725 tablet_service.cc:1514] Processing DeleteTablet for tablet c57f42058228469c8922b522750eca4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:40 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:56094
I20260620 23:16:40.142529 2457 tablet_service.cc:1514] Processing DeleteTablet for tablet c57f42058228469c8922b522750eca4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:40 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:58900
I20260620 23:16:40.142750 2832 tablet_replica.cc:330] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef: stopping tablet replica
I20260620 23:16:40.142719 2591 tablet_service.cc:1514] Processing DeleteTablet for tablet c57f42058228469c8922b522750eca4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:40 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:37244
I20260620 23:16:40.142874 2832 raft_consensus.cc:2229] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:16:40.143007 2833 tablet_replica.cc:330] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9: stopping tablet replica
I20260620 23:16:40.143090 2833 raft_consensus.cc:2229] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:16:40.143155 2832 raft_consensus.cc:2258] T c57f42058228469c8922b522750eca4b P 04b3253a7c8b4e4e9382897577bc1cef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:40.143253 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 2394
I20260620 23:16:40.143213 2833 raft_consensus.cc:2258] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:40.144204 2833 ts_tablet_manager.cc:1905] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:16:40.145905 2833 ts_tablet_manager.cc:1918] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:16:40.145954 2833 log.cc:1196] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/c57f42058228469c8922b522750eca4b
I20260620 23:16:40.146143 2833 ts_tablet_manager.cc:1939] T c57f42058228469c8922b522750eca4b P 16a2c5fe9fcd4ccaa31008f35fced6f9: Deleting consensus metadata
I20260620 23:16:40.146286 2831 tablet_replica.cc:330] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf: stopping tablet replica
I20260620 23:16:40.146422 2831 raft_consensus.cc:2229] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:16:40.146502 2831 raft_consensus.cc:2258] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:40.146631 2318 catalog_manager.cc:4841] TS 16a2c5fe9fcd4ccaa31008f35fced6f9 (127.16.79.2:43739): tablet c57f42058228469c8922b522750eca4b (table test-table [id=f9ae7e19e08b4fac974c89870bcc0046]) successfully deleted
I20260620 23:16:40.146895 2831 ts_tablet_manager.cc:1905] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:16:40.148118 2831 ts_tablet_manager.cc:1918] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:16:40.148202 2831 log.cc:1196] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/c57f42058228469c8922b522750eca4b
I20260620 23:16:40.148478 2831 ts_tablet_manager.cc:1939] T c57f42058228469c8922b522750eca4b P 13976fdd9a11464da3e3d775e968e9bf: Deleting consensus metadata
I20260620 23:16:40.148880 2317 catalog_manager.cc:4841] TS 13976fdd9a11464da3e3d775e968e9bf (127.16.79.3:35233): tablet c57f42058228469c8922b522750eca4b (table test-table [id=f9ae7e19e08b4fac974c89870bcc0046]) successfully deleted
W20260620 23:16:40.152534 2316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:36631 (error 108)
I20260620 23:16:40.152637 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 2528
I20260620 23:16:40.153810 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.153683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:36631 (local address 127.0.0.1:58914)
0620 23:16:40.153739 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.153775 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:36631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 23:16:40.153988 2316 catalog_manager.cc:4584] TS 04b3253a7c8b4e4e9382897577bc1cef (127.16.79.1:36631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c57f42058228469c8922b522750eca4b: Network error: Client connection negotiation failed: client connection to 127.16.79.1:36631: connect: Connection refused (error 111)
I20260620 23:16:40.157430 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 2662
I20260620 23:16:40.162352 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 2300
2026-06-20T23:16:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2141 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T23:16:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:40Z Disabled control of system clock
I20260620 23:16:40.192514 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:34105
--webserver_interface=127.16.79.62
--webserver_port=0
--builtin_ntp_servers=127.16.79.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:34105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:16:40.280900 2838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:40.281193 2838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:40.281219 2838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:40.282853 2838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:40.282914 2838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:40.282929 2838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:40.282943 2838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:40.282958 2838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:40.284539 2838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:34105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:34105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:40.284888 2838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:40.285274 2838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:40.290561 2846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:40.290560 2843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:40.290691 2844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:40.291364 2838 server_base.cc:1034] running on GCE node
I20260620 23:16:40.291661 2838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:40.291980 2838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:40.293159 2838 hybrid_clock.cc:648] HybridClock initialized: now 1781997400293189 us; error 97 us; skew 500 ppm
I20260620 23:16:40.294464 2838 webserver.cc:460] Webserver started at http://127.16.79.62:40571/ using document root <none> and password file <none>
I20260620 23:16:40.294731 2838 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:40.294801 2838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:40.294961 2838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:40.296044 2838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "8a9404c2a1ae4d25837f7cec8d372655"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-pswb"
server_key: "b5253e346a34ecdd3e7e40a7d7802863"
server_key_iv: "61bad3c6c68f3b5f899858bbd8d9f9e9"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.296407 2838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "8a9404c2a1ae4d25837f7cec8d372655"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-pswb"
server_key: "b5253e346a34ecdd3e7e40a7d7802863"
server_key_iv: "61bad3c6c68f3b5f899858bbd8d9f9e9"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.297796 2838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:16:40.298739 2852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:40.298967 2838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:40.299072 2838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "8a9404c2a1ae4d25837f7cec8d372655"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-pswb"
server_key: "b5253e346a34ecdd3e7e40a7d7802863"
server_key_iv: "61bad3c6c68f3b5f899858bbd8d9f9e9"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.299156 2838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:40.310101 2838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:40.310380 2838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:40.310523 2838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:40.314304 2838 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:34105
I20260620 23:16:40.314366 2904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:34105 every 8 connection(s)
I20260620 23:16:40.314695 2838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:40.315631 2905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:40.316998 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 2838
I20260620 23:16:40.317123 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:40.317327 16700 external_mini_cluster.cc:1428] Setting key 9f0f141e401ec6f714546a8dfdaa0249
I20260620 23:16:40.318611 2905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655: Bootstrap starting.
I20260620 23:16:40.319335 2905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:40.319671 2905 log.cc:824] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:40.320772 2905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655: No bootstrap required, opened a new log
I20260620 23:16:40.323020 2905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9404c2a1ae4d25837f7cec8d372655" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34105 } }
I20260620 23:16:40.323197 2905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:40.323238 2905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9404c2a1ae4d25837f7cec8d372655, State: Initialized, Role: FOLLOWER
I20260620 23:16:40.323398 2905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9404c2a1ae4d25837f7cec8d372655" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34105 } }
I20260620 23:16:40.323488 2905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:40.323513 2905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:40.323591 2905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:40.323822 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.317840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.16.79.62:34105)
0620 23:16:40.318024 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.318030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.318064 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:16:40.318458 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.318464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.318774 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.318994 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:40.319254 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.320410 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.321608 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.322046 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.322154 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.322858 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:40.322861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:40.322862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:40.323021 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:40.323051 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:40.323061 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:40.323113 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.323322 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.323417 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260620 23:16:40.324244 2905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9404c2a1ae4d25837f7cec8d372655" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34105 } }
I20260620 23:16:40.324361 2905 leader_election.cc:302] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a9404c2a1ae4d25837f7cec8d372655; no voters:
I20260620 23:16:40.324532 2905 leader_election.cc:288] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:40.324579 2910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:40.324745 2910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [term 1 LEADER]: Becoming Leader. State: Replica: 8a9404c2a1ae4d25837f7cec8d372655, State: Running, Role: LEADER
I20260620 23:16:40.324812 2905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:40.324836 2910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9404c2a1ae4d25837f7cec8d372655" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34105 } }
I20260620 23:16:40.325240 2911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a9404c2a1ae4d25837f7cec8d372655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9404c2a1ae4d25837f7cec8d372655" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34105 } } }
I20260620 23:16:40.325259 2912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a9404c2a1ae4d25837f7cec8d372655. Latest consensus state: current_term: 1 leader_uuid: "8a9404c2a1ae4d25837f7cec8d372655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9404c2a1ae4d25837f7cec8d372655" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 34105 } } }
I20260620 23:16:40.325440 2912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:40.325397 2911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:40.325769 2919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:40.326268 2919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:40.327946 2919 catalog_manager.cc:1347] Generated new cluster ID: ff3a3133fb6343548df091aa0fa5ed02
I20260620 23:16:40.328058 2919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:40.335227 2919 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:40.335894 2919 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:40.344128 2919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655: Generated new TSK 0
I20260620 23:16:40.344290 2919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:40.348483 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34105
--builtin_ntp_servers=127.16.79.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:16:40.439535 2929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:40.439764 2929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:40.439800 2929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:40.440975 2929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:40.441033 2929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:40.441079 2929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:16:40.442411 2929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:40.442656 2929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:40.442945 2929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:40.448549 2934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:40.448596 2937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:40.448562 2935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:40.448897 2929 server_base.cc:1034] running on GCE node
I20260620 23:16:40.449165 2929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:40.449402 2929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:40.450560 2929 hybrid_clock.cc:648] HybridClock initialized: now 1781997400450543 us; error 35 us; skew 500 ppm
I20260620 23:16:40.451782 2929 webserver.cc:460] Webserver started at http://127.16.79.1:37479/ using document root <none> and password file <none>
I20260620 23:16:40.452025 2929 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:40.452095 2929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:40.452229 2929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:40.453327 2929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "6e4cc2ea014b45a0924e6bfbda0c6426"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-pswb"
server_key: "f29dd28505c34f66ee36d6d84401cd2d"
server_key_iv: "62691d76e2ca9560bf6d1ece34722819"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.453688 2929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "6e4cc2ea014b45a0924e6bfbda0c6426"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-pswb"
server_key: "f29dd28505c34f66ee36d6d84401cd2d"
server_key_iv: "62691d76e2ca9560bf6d1ece34722819"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.455153 2929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:16:40.455904 2943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:40.456066 2929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:16:40.456161 2929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "6e4cc2ea014b45a0924e6bfbda0c6426"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-pswb"
server_key: "f29dd28505c34f66ee36d6d84401cd2d"
server_key_iv: "62691d76e2ca9560bf6d1ece34722819"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.456241 2929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:40.494143 2929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:40.494488 2929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:40.494645 2929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:40.494933 2929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:40.495267 2929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:40.495324 2929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:40.495357 2929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:40.495385 2929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:40.501137 2929 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:39685
I20260620 23:16:40.501201 3056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:39685 every 8 connection(s)
I20260620 23:16:40.501569 2929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:16:40.505142 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 2929
I20260620 23:16:40.505255 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:16:40.505456 16700 external_mini_cluster.cc:1428] Setting key d8b7f8af2fe9654cc41cfcf26e2be707
I20260620 23:16:40.506510 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34105
--builtin_ntp_servers=127.16.79.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:16:40.507566 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.502313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:58925 (local address 127.16.79.62:34105)
0620 23:16:40.502386 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.502388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.502678 (+ 290us) server_negotiation.cc:407] Connection header received
0620 23:16:40.503230 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.503231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.503242 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.503291 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:40.503976 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.504490 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.505940 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.506287 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.506367 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.507313 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:40.507315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:40.507315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:40.507319 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:40.507333 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:40.507336 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:40.507379 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.507464 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.507491 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:16:40.507784 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.502314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34105 (local address 127.16.79.1:58925)
0620 23:16:40.502558 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.502597 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.503121 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.503383 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.503388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.503403 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:40.503876 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.503880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.504587 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.504588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.505839 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.505842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.506375 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.506377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.506519 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.507157 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:16:40.507157 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:40.507184 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:16:40.507206 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:40.507390 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:40.507395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:40.507398 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.507465 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.507563 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
I20260620 23:16:40.508311 3057 heartbeater.cc:347] Connected to a master server at 127.16.79.62:34105
I20260620 23:16:40.508410 3057 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:40.508581 3057 heartbeater.cc:511] Master 127.16.79.62:34105 requested a full tablet report, sending...
I20260620 23:16:40.509097 2869 ts_manager.cc:194] Registered new tserver with Master: 6e4cc2ea014b45a0924e6bfbda0c6426 (127.16.79.1:39685)
I20260620 23:16:40.510002 2869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.79.1:58925
W20260620 23:16:40.589033 3060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:40.589233 3060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:40.589254 3060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:40.590294 3060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:40.590328 3060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:40.590353 3060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:16:40.591568 3060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:40.591759 3060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:40.591986 3060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:40.597173 3066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:40.597241 3068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:40.597191 3065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:40.597797 3060 server_base.cc:1034] running on GCE node
I20260620 23:16:40.598055 3060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:40.598296 3060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:40.599481 3060 hybrid_clock.cc:648] HybridClock initialized: now 1781997400599409 us; error 119 us; skew 500 ppm
I20260620 23:16:40.600783 3060 webserver.cc:460] Webserver started at http://127.16.79.2:36749/ using document root <none> and password file <none>
I20260620 23:16:40.601034 3060 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:40.601106 3060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:40.601238 3060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:40.602406 3060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "a08e347d38024fd09dfcffda1ad513f9"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-pswb"
server_key: "c438ab585fdf066ff7a690cc6591d3eb"
server_key_iv: "382f346d8537381489c27714eff3cc36"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.602783 3060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "a08e347d38024fd09dfcffda1ad513f9"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-pswb"
server_key: "c438ab585fdf066ff7a690cc6591d3eb"
server_key_iv: "382f346d8537381489c27714eff3cc36"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.604156 3060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:16:40.605069 3074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:40.605239 3060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:16:40.605338 3060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "a08e347d38024fd09dfcffda1ad513f9"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-pswb"
server_key: "c438ab585fdf066ff7a690cc6591d3eb"
server_key_iv: "382f346d8537381489c27714eff3cc36"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.605414 3060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:40.629405 3060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:40.629706 3060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:40.629859 3060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:40.630095 3060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:40.630420 3060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:40.630477 3060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:40.630513 3060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:40.630540 3060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:40.636322 3060 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:38183
I20260620 23:16:40.636380 3187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:38183 every 8 connection(s)
I20260620 23:16:40.636674 3060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:16:40.642055 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 3060
I20260620 23:16:40.642166 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:16:40.642371 16700 external_mini_cluster.cc:1428] Setting key ee12817275f52c45dd8cbae64fbbf9c1
I20260620 23:16:40.642752 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.637456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:33275 (local address 127.16.79.62:34105)
0620 23:16:40.637638 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.637642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.638023 (+ 381us) server_negotiation.cc:407] Connection header received
0620 23:16:40.638566 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.638567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.638579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.638636 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:40.639326 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.639820 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.641194 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.641391 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.641488 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.642430 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:40.642432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:40.642432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:40.642438 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:40.642455 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:40.642460 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:40.642522 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.642635 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.642668 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 23:16:40.642894 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.637423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34105 (local address 127.16.79.2:33275)
0620 23:16:40.637762 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.637816 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.638456 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.638732 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.638735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.638744 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:40.639225 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.639228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.639917 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.639918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.640907 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.640913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.641514 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.641516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.641649 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.642268 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:16:40.642271 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:40.642312 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:16:40.642335 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:40.642536 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:40.642541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:40.642544 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.642612 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.642696 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":98,"threads_started":1}
I20260620 23:16:40.643352 3188 heartbeater.cc:347] Connected to a master server at 127.16.79.62:34105
I20260620 23:16:40.643433 3188 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:40.643537 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34105
--builtin_ntp_servers=127.16.79.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:16:40.643611 3188 heartbeater.cc:511] Master 127.16.79.62:34105 requested a full tablet report, sending...
I20260620 23:16:40.643993 2869 ts_manager.cc:194] Registered new tserver with Master: a08e347d38024fd09dfcffda1ad513f9 (127.16.79.2:38183)
I20260620 23:16:40.644505 2869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.79.2:33275
W20260620 23:16:40.728798 3191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:40.729007 3191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:40.729029 3191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:40.730165 3191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:40.730207 3191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:40.730232 3191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:16:40.731539 3191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:34105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:40.731769 3191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:40.732079 3191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:40.737071 3196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:40.737072 3199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:40.737108 3197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:40.737164 3191 server_base.cc:1034] running on GCE node
I20260620 23:16:40.737500 3191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:40.737741 3191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:40.738981 3191 hybrid_clock.cc:648] HybridClock initialized: now 1781997400738975 us; error 48 us; skew 500 ppm
I20260620 23:16:40.740685 3191 webserver.cc:460] Webserver started at http://127.16.79.3:39537/ using document root <none> and password file <none>
I20260620 23:16:40.741017 3191 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:40.741086 3191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:40.741222 3191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:40.742295 3191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "b0a6f6cca9c8427881f010039ebe2312"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-pswb"
server_key: "afea4dbdbb37774b87472ebf1a4ca7f8"
server_key_iv: "00f6b5f5b83c8fcef9edd87f8a31af2a"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.742658 3191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "b0a6f6cca9c8427881f010039ebe2312"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-pswb"
server_key: "afea4dbdbb37774b87472ebf1a4ca7f8"
server_key_iv: "00f6b5f5b83c8fcef9edd87f8a31af2a"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.744083 3191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:16:40.744891 3205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:40.745046 3191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:16:40.745133 3191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "b0a6f6cca9c8427881f010039ebe2312"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-pswb"
server_key: "afea4dbdbb37774b87472ebf1a4ca7f8"
server_key_iv: "00f6b5f5b83c8fcef9edd87f8a31af2a"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.745182 3191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:40.755055 3191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:40.755281 3191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:40.755386 3191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:40.755556 3191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:40.755854 3191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:40.755893 3191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:40.755915 3191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:40.755929 3191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:40.762588 3191 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:43413
I20260620 23:16:40.762641 3318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:43413 every 8 connection(s)
I20260620 23:16:40.762961 3191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:16:40.768582 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 3191
I20260620 23:16:40.768632 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.763646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:37177 (local address 127.16.79.62:34105)
0620 23:16:40.763723 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.763725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.763930 (+ 205us) server_negotiation.cc:407] Connection header received
0620 23:16:40.764423 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.764424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.764433 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.764485 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:40.765287 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.765846 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.766956 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.767161 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.767244 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.768283 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:40.768284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:40.768285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:40.768288 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:40.768301 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:40.768304 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:40.768363 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.768495 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.768522 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:16:40.768712 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:16:40.768936 16700 external_mini_cluster.cc:1428] Setting key 85c06797911d5d61ad6d049530668dd2
I20260620 23:16:40.768921 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.763587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:34105 (local address 127.16.79.3:37177)
0620 23:16:40.763819 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.763855 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.764321 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.764535 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.764539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.764547 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:40.765119 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.765125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.765973 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.765974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.766842 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.766845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.767320 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.767322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.767510 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.768113 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:16:40.768114 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:40.768145 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:16:40.768169 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:40.768397 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:40.768401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:40.768404 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.768462 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.768548 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260620 23:16:40.769588 3319 heartbeater.cc:347] Connected to a master server at 127.16.79.62:34105
I20260620 23:16:40.769687 3319 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:40.769905 3319 heartbeater.cc:511] Master 127.16.79.62:34105 requested a full tablet report, sending...
I20260620 23:16:40.770252 2868 ts_manager.cc:194] Registered new tserver with Master: b0a6f6cca9c8427881f010039ebe2312 (127.16.79.3:43413)
I20260620 23:16:40.770572 2868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.79.3:37177
I20260620 23:16:40.781152 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:16:40.785789 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.782749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.16.79.62:34105)
0620 23:16:40.782809 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.782811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.782942 (+ 131us) server_negotiation.cc:407] Connection header received
0620 23:16:40.783005 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.783006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.783016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.783071 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:40.783235 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.783724 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.784482 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.784687 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.784776 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.785492 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:40.785494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:40.785494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:40.785498 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:40.785511 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:40.785514 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:40.785614 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.785681 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.785691 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:16:40.787436 2868 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:16:40.788928 2868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:16:40.795437 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.791638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.16.79.1:39685)
0620 23:16:40.791834 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.791839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.791850 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:16:40.791891 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.791893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.791909 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.792006 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:40.792161 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.793317 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.794566 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.795048 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.795139 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.795227 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.795322 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.795357 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":129,"threads_started":1}
I20260620 23:16:40.795430 3328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.791368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:39685 (local address 127.0.0.1:39294)
0620 23:16:40.791544 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.791572 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.791634 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.792014 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.792017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.792018 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:40.792074 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.792076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.793404 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.793405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.794460 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.794462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.795140 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.795140 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.795222 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.795236 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.795308 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.795357 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260620 23:16:40.795557 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.791368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:38183 (local address 127.0.0.1:60058)
0620 23:16:40.791537 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.791571 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.791635 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.792119 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.792120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.792122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:40.792166 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.792167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.793449 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.793450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.794404 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.794406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.795300 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.795302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.795364 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.795369 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.795400 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.795443 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260620 23:16:40.795760 3329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.791381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:43413 (local address 127.0.0.1:49488)
0620 23:16:40.791855 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.791865 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.791920 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.792123 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.792124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.792126 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:40.792171 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.792172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.793220 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.793222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.794470 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.794472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.795385 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.795386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.795438 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.795441 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.795670 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.795706 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260620 23:16:40.795995 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.791704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.16.79.3:43413)
0620 23:16:40.791875 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.791879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.791911 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:16:40.791980 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.791981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.791994 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.792064 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:40.792271 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.793127 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.794531 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.795195 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.795277 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.795354 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.795875 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.795923 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260620 23:16:40.796162 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.791545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.16.79.2:38183)
0620 23:16:40.791915 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.791919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.791928 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:16:40.791967 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.791969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.791985 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.792100 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:40.792261 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.793349 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.794491 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.795195 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.795926 (+ 731us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.796012 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.796036 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.796074 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":77,"threads_started":1}
I20260620 23:16:40.796105 2991 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0e5a35e254460bbc9cfedc3cf4c987 (DEFAULT_TABLE table=test-table [id=b01286eb99264ec3a5548c404055e5f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:40.796461 2991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0e5a35e254460bbc9cfedc3cf4c987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:40.797276 3122 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0e5a35e254460bbc9cfedc3cf4c987 (DEFAULT_TABLE table=test-table [id=b01286eb99264ec3a5548c404055e5f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:40.797569 3122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0e5a35e254460bbc9cfedc3cf4c987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:40.797494 3253 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0e5a35e254460bbc9cfedc3cf4c987 (DEFAULT_TABLE table=test-table [id=b01286eb99264ec3a5548c404055e5f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:40.797799 3253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0e5a35e254460bbc9cfedc3cf4c987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:40.799875 3333 tablet_bootstrap.cc:492] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426: Bootstrap starting.
I20260620 23:16:40.800145 3334 tablet_bootstrap.cc:492] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9: Bootstrap starting.
I20260620 23:16:40.800619 3333 tablet_bootstrap.cc:654] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:40.800805 3334 tablet_bootstrap.cc:654] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:40.801051 3333 log.cc:824] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:40.801122 3334 log.cc:824] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:40.801095 3335 tablet_bootstrap.cc:492] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312: Bootstrap starting.
I20260620 23:16:40.801802 3333 tablet_bootstrap.cc:492] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426: No bootstrap required, opened a new log
I20260620 23:16:40.801888 3335 tablet_bootstrap.cc:654] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:40.801882 3333 ts_tablet_manager.cc:1397] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 23:16:40.801934 3334 tablet_bootstrap.cc:492] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9: No bootstrap required, opened a new log
I20260620 23:16:40.802002 3334 ts_tablet_manager.cc:1397] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 23:16:40.802183 3335 log.cc:824] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:40.803004 3335 tablet_bootstrap.cc:492] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312: No bootstrap required, opened a new log
I20260620 23:16:40.803071 3335 ts_tablet_manager.cc:1397] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:16:40.804036 3334 raft_consensus.cc:348] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08e347d38024fd09dfcffda1ad513f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38183 } } peers { permanent_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39685 } } peers { permanent_uuid: "b0a6f6cca9c8427881f010039ebe2312" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43413 } }
I20260620 23:16:40.804190 3334 raft_consensus.cc:374] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:40.804247 3334 raft_consensus.cc:729] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a08e347d38024fd09dfcffda1ad513f9, State: Initialized, Role: FOLLOWER
I20260620 23:16:40.804368 3334 consensus_queue.cc:260] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08e347d38024fd09dfcffda1ad513f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38183 } } peers { permanent_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39685 } } peers { permanent_uuid: "b0a6f6cca9c8427881f010039ebe2312" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43413 } }
I20260620 23:16:40.804550 3333 raft_consensus.cc:348] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08e347d38024fd09dfcffda1ad513f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38183 } } peers { permanent_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39685 } } peers { permanent_uuid: "b0a6f6cca9c8427881f010039ebe2312" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43413 } }
I20260620 23:16:40.804735 3333 raft_consensus.cc:374] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:40.804781 3333 raft_consensus.cc:729] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4cc2ea014b45a0924e6bfbda0c6426, State: Initialized, Role: FOLLOWER
I20260620 23:16:40.804858 3334 ts_tablet_manager.cc:1428] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:16:40.804911 3333 consensus_queue.cc:260] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08e347d38024fd09dfcffda1ad513f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38183 } } peers { permanent_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39685 } } peers { permanent_uuid: "b0a6f6cca9c8427881f010039ebe2312" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43413 } }
I20260620 23:16:40.804659 3188 heartbeater.cc:503] Master 127.16.79.62:34105 was elected leader, sending a full tablet report...
I20260620 23:16:40.805143 3333 ts_tablet_manager.cc:1428] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 23:16:40.805258 3057 heartbeater.cc:503] Master 127.16.79.62:34105 was elected leader, sending a full tablet report...
I20260620 23:16:40.805635 3335 raft_consensus.cc:348] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08e347d38024fd09dfcffda1ad513f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38183 } } peers { permanent_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39685 } } peers { permanent_uuid: "b0a6f6cca9c8427881f010039ebe2312" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43413 } }
I20260620 23:16:40.805773 3335 raft_consensus.cc:374] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:40.805826 3335 raft_consensus.cc:729] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a6f6cca9c8427881f010039ebe2312, State: Initialized, Role: FOLLOWER
I20260620 23:16:40.805933 3335 consensus_queue.cc:260] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08e347d38024fd09dfcffda1ad513f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38183 } } peers { permanent_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39685 } } peers { permanent_uuid: "b0a6f6cca9c8427881f010039ebe2312" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43413 } }
I20260620 23:16:40.806169 3335 ts_tablet_manager.cc:1428] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:16:40.806304 3319 heartbeater.cc:503] Master 127.16.79.62:34105 was elected leader, sending a full tablet report...
I20260620 23:16:40.842834 3339 raft_consensus.cc:482] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:16:40.843003 3339 raft_consensus.cc:504] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08e347d38024fd09dfcffda1ad513f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38183 } } peers { permanent_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39685 } } peers { permanent_uuid: "b0a6f6cca9c8427881f010039ebe2312" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43413 } }
I20260620 23:16:40.843386 3339 leader_election.cc:288] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e4cc2ea014b45a0924e6bfbda0c6426 (127.16.79.1:39685), b0a6f6cca9c8427881f010039ebe2312 (127.16.79.3:43413)
I20260620 23:16:40.846565 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.843645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:48041 (local address 127.16.79.1:39685)
0620 23:16:40.843698 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.843700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.843711 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:16:40.843774 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.843775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.843795 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.843865 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:40.844014 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.844537 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.845729 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.846249 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.846300 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.846307 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.846484 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.846515 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:16:40.846593 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.843550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:39685 (local address 127.16.79.2:48041)
0620 23:16:40.843645 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.843657 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.843699 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.843869 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.843870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.843872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:40.843945 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.843947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.844620 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.844621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.845645 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.845648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.846327 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.846328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.846406 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.846410 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.846474 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.846516 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":1}
I20260620 23:16:40.846954 3011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0e5a35e254460bbc9cfedc3cf4c987" candidate_uuid: "a08e347d38024fd09dfcffda1ad513f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426" is_pre_election: true
I20260620 23:16:40.847110 3011 raft_consensus.cc:2454] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a08e347d38024fd09dfcffda1ad513f9 in term 0.
I20260620 23:16:40.847121 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.843617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:43413 (local address 127.16.79.2:45749)
0620 23:16:40.843780 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.843799 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.843904 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.844082 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.844083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.844085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:40.844164 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.844166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.845177 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.845179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.846069 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.846071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.846825 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.846826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.846917 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.846920 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.847005 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.847044 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20260620 23:16:40.847121 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.843671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:45749 (local address 127.16.79.3:43413)
0620 23:16:40.843727 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.843729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.843888 (+ 159us) server_negotiation.cc:407] Connection header received
0620 23:16:40.843980 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.843980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.843991 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.844064 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:40.844238 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.845086 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.846160 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.846752 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.846820 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.846829 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.847015 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.847055 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:16:40.847342 3076 leader_election.cc:302] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e4cc2ea014b45a0924e6bfbda0c6426, a08e347d38024fd09dfcffda1ad513f9; no voters:
I20260620 23:16:40.847448 3339 raft_consensus.cc:2781] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:16:40.847503 3339 raft_consensus.cc:482] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:16:40.847472 3273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0e5a35e254460bbc9cfedc3cf4c987" candidate_uuid: "a08e347d38024fd09dfcffda1ad513f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a6f6cca9c8427881f010039ebe2312" is_pre_election: true
I20260620 23:16:40.847520 3339 raft_consensus.cc:3037] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:40.847613 3273 raft_consensus.cc:2454] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a08e347d38024fd09dfcffda1ad513f9 in term 0.
I20260620 23:16:40.848281 3339 raft_consensus.cc:504] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08e347d38024fd09dfcffda1ad513f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38183 } } peers { permanent_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39685 } } peers { permanent_uuid: "b0a6f6cca9c8427881f010039ebe2312" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43413 } }
I20260620 23:16:40.848433 3339 leader_election.cc:288] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [CANDIDATE]: Term 1 election: Requested vote from peers 6e4cc2ea014b45a0924e6bfbda0c6426 (127.16.79.1:39685), b0a6f6cca9c8427881f010039ebe2312 (127.16.79.3:43413)
I20260620 23:16:40.848609 3273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0e5a35e254460bbc9cfedc3cf4c987" candidate_uuid: "a08e347d38024fd09dfcffda1ad513f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a6f6cca9c8427881f010039ebe2312"
I20260620 23:16:40.848637 3011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0e5a35e254460bbc9cfedc3cf4c987" candidate_uuid: "a08e347d38024fd09dfcffda1ad513f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426"
I20260620 23:16:40.848717 3273 raft_consensus.cc:3037] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:40.848739 3011 raft_consensus.cc:3037] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:40.849668 3011 raft_consensus.cc:2454] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a08e347d38024fd09dfcffda1ad513f9 in term 1.
I20260620 23:16:40.849668 3273 raft_consensus.cc:2454] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a08e347d38024fd09dfcffda1ad513f9 in term 1.
I20260620 23:16:40.849850 3076 leader_election.cc:302] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e4cc2ea014b45a0924e6bfbda0c6426, a08e347d38024fd09dfcffda1ad513f9; no voters:
I20260620 23:16:40.849943 3339 raft_consensus.cc:2781] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:40.850072 3339 raft_consensus.cc:686] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 1 LEADER]: Becoming Leader. State: Replica: a08e347d38024fd09dfcffda1ad513f9, State: Running, Role: LEADER
I20260620 23:16:40.850173 3339 consensus_queue.cc:237] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08e347d38024fd09dfcffda1ad513f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38183 } } peers { permanent_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39685 } } peers { permanent_uuid: "b0a6f6cca9c8427881f010039ebe2312" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43413 } }
I20260620 23:16:40.851029 2868 catalog_manager.cc:5495] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a08e347d38024fd09dfcffda1ad513f9 (127.16.79.2). New cstate: current_term: 1 leader_uuid: "a08e347d38024fd09dfcffda1ad513f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08e347d38024fd09dfcffda1ad513f9" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 38183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0a6f6cca9c8427881f010039ebe2312" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43413 } health_report { overall_health: UNKNOWN } } }
I20260620 23:16:40.871932 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.869065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.16.79.2:38183)
0620 23:16:40.869115 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.869117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.869125 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:16:40.869195 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.869195 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.869212 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.869285 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:40.869527 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.870043 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.870963 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.871156 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.871216 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.871524 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:40.871649 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:40.871755 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.871819 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.871834 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:16:40.873986 3273 raft_consensus.cc:1261] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312 [term 1 FOLLOWER]: Refusing update from remote peer a08e347d38024fd09dfcffda1ad513f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:40.874018 3011 raft_consensus.cc:1261] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426 [term 1 FOLLOWER]: Refusing update from remote peer a08e347d38024fd09dfcffda1ad513f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:40.874367 3339 consensus_queue.cc:1035] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a6f6cca9c8427881f010039ebe2312" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 43413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:40.874531 3343 consensus_queue.cc:1035] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e4cc2ea014b45a0924e6bfbda0c6426" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 39685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:40.876960 3348 mvcc.cc:205] Tried to move back new op lower bound from 7299061353976913920 to 7299061353882796032. Current Snapshot: MvccSnapshot[applied={T|T < 7299061353976913920}]
I20260620 23:16:40.877485 3347 mvcc.cc:205] Tried to move back new op lower bound from 7299061353976913920 to 7299061353882796032. Current Snapshot: MvccSnapshot[applied={T|T < 7299061353976913920}]
I20260620 23:16:40.879144 2868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:16:40.879248 2868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:16:40.880031 2868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a9404c2a1ae4d25837f7cec8d372655: Sending DeleteTablet for 3 replicas of tablet 4a0e5a35e254460bbc9cfedc3cf4c987
I20260620 23:16:40.880394 3122 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0e5a35e254460bbc9cfedc3cf4c987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:40 UTC) from {username='slave'} at 127.0.0.1:60058
I20260620 23:16:40.880393 3253 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0e5a35e254460bbc9cfedc3cf4c987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:40 UTC) from {username='slave'} at 127.0.0.1:49488
I20260620 23:16:40.880604 2991 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0e5a35e254460bbc9cfedc3cf4c987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:40 UTC) from {username='slave'} at 127.0.0.1:39294
I20260620 23:16:40.880681 3356 tablet_replica.cc:330] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312: stopping tablet replica
I20260620 23:16:40.880813 3356 raft_consensus.cc:2229] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:16:40.880821 3358 tablet_replica.cc:330] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426: stopping tablet replica
I20260620 23:16:40.880895 3358 raft_consensus.cc:2229] T 4a0e5a35e254460bbc9cfedc3cf4c987 P 6e4cc2ea014b45a0924e6bfbda0c6426 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:16:40.880920 3356 raft_consensus.cc:2258] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:40.880982 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 2929
I20260620 23:16:40.881598 3357 tablet_replica.cc:330] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9: stopping tablet replica
I20260620 23:16:40.881695 3357 raft_consensus.cc:2229] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:16:40.881807 3357 raft_consensus.cc:2258] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:40.882217 3357 ts_tablet_manager.cc:1905] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:16:40.882644 3356 ts_tablet_manager.cc:1905] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:16:40.883958 3357 ts_tablet_manager.cc:1918] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:16:40.884032 3357 log.cc:1196] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/4a0e5a35e254460bbc9cfedc3cf4c987
I20260620 23:16:40.884280 3356 ts_tablet_manager.cc:1918] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:16:40.884301 3357 ts_tablet_manager.cc:1939] T 4a0e5a35e254460bbc9cfedc3cf4c987 P a08e347d38024fd09dfcffda1ad513f9: Deleting consensus metadata
I20260620 23:16:40.884351 3356 log.cc:1196] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/4a0e5a35e254460bbc9cfedc3cf4c987
I20260620 23:16:40.884578 3356 ts_tablet_manager.cc:1939] T 4a0e5a35e254460bbc9cfedc3cf4c987 P b0a6f6cca9c8427881f010039ebe2312: Deleting consensus metadata
I20260620 23:16:40.884672 2855 catalog_manager.cc:4841] TS a08e347d38024fd09dfcffda1ad513f9 (127.16.79.2:38183): tablet 4a0e5a35e254460bbc9cfedc3cf4c987 (table test-table [id=b01286eb99264ec3a5548c404055e5f3]) successfully deleted
I20260620 23:16:40.885004 2853 catalog_manager.cc:4841] TS b0a6f6cca9c8427881f010039ebe2312 (127.16.79.3:43413): tablet 4a0e5a35e254460bbc9cfedc3cf4c987 (table test-table [id=b01286eb99264ec3a5548c404055e5f3]) successfully deleted
W20260620 23:16:40.886713 2854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:39685 (error 108)
I20260620 23:16:40.886960 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 3060
I20260620 23:16:40.887133 3329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.887022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:39685 (local address 127.0.0.1:39302)
0620 23:16:40.887070 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.887103 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:39685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 23:16:40.888167 2854 catalog_manager.cc:4584] TS 6e4cc2ea014b45a0924e6bfbda0c6426 (127.16.79.1:39685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a0e5a35e254460bbc9cfedc3cf4c987: Network error: Client connection negotiation failed: client connection to 127.16.79.1:39685: connect: Connection refused (error 111)
I20260620 23:16:40.891695 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 3191
I20260620 23:16:40.897300 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 2838
2026-06-20T23:16:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (733 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:40 dist-test-slave-pswb krb5kdc[3362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:40 dist-test-slave-pswb krb5kdc[3362](info): set up 2 sockets
Jun 20 23:16:40 dist-test-slave-pswb krb5kdc[3362](info): commencing operation
krb5kdc: starting...
W20260620 23:16:41.953655 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:42 dist-test-slave-pswb krb5kdc[3362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:16:42.076089 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:40405
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:40405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:42.160312 3375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:42.160506 3375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:42.160528 3375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:42.161676 3375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:42.161718 3375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:42.161733 3375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:42.161746 3375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:42.161758 3375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:42.163210 3375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:40405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:40405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:42.163461 3375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:42.163690 3375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:42.168466 3380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:42.168426 3383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:42.168562 3381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:42.168742 3375 server_base.cc:1034] running on GCE node
I20260620 23:16:42.169072 3375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:42.169404 3375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:42.170593 3375 hybrid_clock.cc:648] HybridClock initialized: now 1781997402170562 us; error 44 us; skew 500 ppm
Jun 20 23:16:42 dist-test-slave-pswb krb5kdc[3362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:42.173740 3375 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:16:42.174293 3375 webserver.cc:460] Webserver started at http://127.16.79.62:43641/ using document root <none> and password file <none>
I20260620 23:16:42.174548 3375 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:42.174609 3375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:42.174750 3375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:42.175947 3375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "355fbeb71bf343aa9921facaa434df96"
format_stamp: "Formatted at 2026-06-20 23:16:42 on dist-test-slave-pswb"
server_key: "ce0a778f45e8e370986617cdaf4822bb"
server_key_iv: "72170cc214f01fd54cf8c0424e30f4e9"
server_key_version: "encryptionkey@0"
I20260620 23:16:42.176339 3375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "355fbeb71bf343aa9921facaa434df96"
format_stamp: "Formatted at 2026-06-20 23:16:42 on dist-test-slave-pswb"
server_key: "ce0a778f45e8e370986617cdaf4822bb"
server_key_iv: "72170cc214f01fd54cf8c0424e30f4e9"
server_key_version: "encryptionkey@0"
I20260620 23:16:42.177793 3375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:16:42.178596 3390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:42.178889 3375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:16:42.179010 3375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "355fbeb71bf343aa9921facaa434df96"
format_stamp: "Formatted at 2026-06-20 23:16:42 on dist-test-slave-pswb"
server_key: "ce0a778f45e8e370986617cdaf4822bb"
server_key_iv: "72170cc214f01fd54cf8c0424e30f4e9"
server_key_version: "encryptionkey@0"
I20260620 23:16:42.179091 3375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:42.193502 3375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:42.195825 3375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:42.195994 3375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:42.201879 3375 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:40405
I20260620 23:16:42.202028 3442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:40405 every 8 connection(s)
I20260620 23:16:42.202347 3375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:42.203249 3443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:42.205514 3443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96: Bootstrap starting.
I20260620 23:16:42.206071 3443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:42.206367 3443 log.cc:824] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:42.207072 3443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96: No bootstrap required, opened a new log
I20260620 23:16:42.208602 3443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355fbeb71bf343aa9921facaa434df96" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40405 } }
I20260620 23:16:42.208811 3443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:42.208887 3443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355fbeb71bf343aa9921facaa434df96, State: Initialized, Role: FOLLOWER
I20260620 23:16:42.209025 3443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355fbeb71bf343aa9921facaa434df96" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40405 } }
I20260620 23:16:42.209087 3443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:42.209143 3443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:42.209192 3443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:42.209694 3443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355fbeb71bf343aa9921facaa434df96" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40405 } }
I20260620 23:16:42.209775 3443 leader_election.cc:302] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 355fbeb71bf343aa9921facaa434df96; no voters:
I20260620 23:16:42.209950 3443 leader_election.cc:288] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:42.209995 3446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:42.210193 3443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:42.210270 3446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [term 1 LEADER]: Becoming Leader. State: Replica: 355fbeb71bf343aa9921facaa434df96, State: Running, Role: LEADER
I20260620 23:16:42.210362 3446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355fbeb71bf343aa9921facaa434df96" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40405 } }
I20260620 23:16:42.210742 3447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "355fbeb71bf343aa9921facaa434df96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355fbeb71bf343aa9921facaa434df96" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40405 } } }
I20260620 23:16:42.210829 3447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:42.210887 3448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355fbeb71bf343aa9921facaa434df96. Latest consensus state: current_term: 1 leader_uuid: "355fbeb71bf343aa9921facaa434df96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355fbeb71bf343aa9921facaa434df96" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40405 } } }
I20260620 23:16:42.210979 3448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:42.211018 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 3375
I20260620 23:16:42.211035 3456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:42.211135 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:42.211335 16700 external_mini_cluster.cc:1428] Setting key e4205da56fc2c95ab24c3de785620891
I20260620 23:16:42.211552 3456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:42.213824 3456 catalog_manager.cc:1347] Generated new cluster ID: 061fccc888b147a1b5984a382c182397
I20260620 23:16:42.213883 3456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:16:42 dist-test-slave-pswb krb5kdc[3362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:16:42.223860 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.211956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.16.79.62:40405)
0620 23:16:42.212136 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.212148 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.212183 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:16:42.212293 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.212300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.212373 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.213190 (+ 817us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:42.213620 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.214999 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.216006 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.216597 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.216680 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.219488 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:42.219491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:42.219492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:42.219503 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:42.222085 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:42.222426 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:42.222428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:42.222428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:42.222524 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:42.222682 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:42.222683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:42.222683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:42.222965 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:42.223414 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.223501 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.223628 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260620 23:16:42.229866 3456 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:42.230409 3456 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:42.234304 3456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96: Generated new TSK 0
I20260620 23:16:42.234493 3456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.79.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
Entry for principal kudu/127.16.79.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab.
I20260620 23:16:42.271183 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:40405
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:42.352981 3470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:42.353186 3470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:42.353211 3470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:42.354456 3470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:42.354504 3470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:42.354534 3470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:16:42.355955 3470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.1.keytab
--principal=kudu/127.16.79.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:40405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:42.356190 3470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:42.356456 3470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:42.361944 3475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:42.362079 3476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:42.362107 3478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:42.362341 3470 server_base.cc:1034] running on GCE node
I20260620 23:16:42.362525 3470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:42.362718 3470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:42.363929 3470 hybrid_clock.cc:648] HybridClock initialized: now 1781997402363897 us; error 42 us; skew 500 ppm
Jun 20 23:16:42 dist-test-slave-pswb krb5kdc[3362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:42.367043 3470 init.cc:377] Logged in from keytab as kudu/127.16.79.1@KRBTEST.COM (short username kudu)
I20260620 23:16:42.367689 3470 webserver.cc:460] Webserver started at http://127.16.79.1:44043/ using document root <none> and password file <none>
I20260620 23:16:42.367992 3470 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:42.368084 3470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:42.368259 3470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:42.369721 3470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "4f602e444b1c4cb281a43b40cc0391aa"
format_stamp: "Formatted at 2026-06-20 23:16:42 on dist-test-slave-pswb"
server_key: "3ca02a895d8fdb113839bb6153a1dcf1"
server_key_iv: "aa25d80c21e9cf7f1a31134bf0a6c2ae"
server_key_version: "encryptionkey@0"
I20260620 23:16:42.370157 3470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "4f602e444b1c4cb281a43b40cc0391aa"
format_stamp: "Formatted at 2026-06-20 23:16:42 on dist-test-slave-pswb"
server_key: "3ca02a895d8fdb113839bb6153a1dcf1"
server_key_iv: "aa25d80c21e9cf7f1a31134bf0a6c2ae"
server_key_version: "encryptionkey@0"
I20260620 23:16:42.371918 3470 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 23:16:42.372959 3485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:42.373163 3470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:42.373263 3470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "4f602e444b1c4cb281a43b40cc0391aa"
format_stamp: "Formatted at 2026-06-20 23:16:42 on dist-test-slave-pswb"
server_key: "3ca02a895d8fdb113839bb6153a1dcf1"
server_key_iv: "aa25d80c21e9cf7f1a31134bf0a6c2ae"
server_key_version: "encryptionkey@0"
I20260620 23:16:42.373368 3470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:42.386379 3470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:42.388253 3470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:42.388417 3470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:42.388650 3470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:42.389019 3470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:42.389075 3470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:42.389118 3470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:42.389146 3470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:42.395085 3470 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:41983
I20260620 23:16:42.395126 3598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:41983 every 8 connection(s)
I20260620 23:16:42.395448 3470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:16:42.396000 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 3470
I20260620 23:16:42.396140 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:16:42.396347 16700 external_mini_cluster.cc:1428] Setting key 168a00a377a5f13b1213914b798bf6db
Jun 20 23:16:42 dist-test-slave-pswb krb5kdc[3362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.1@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.79.2@KRBTEST.COM; defaulting to no policy
I20260620 23:16:42.406847 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.396640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:56577 (local address 127.16.79.62:40405)
0620 23:16:42.396746 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.396749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.397269 (+ 520us) server_negotiation.cc:407] Connection header received
0620 23:16:42.397979 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.397981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.397998 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.398076 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:42.399299 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.399910 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.401155 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.401408 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.401504 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.403846 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:42.403848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:42.403848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:42.403856 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:42.405326 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:42.405771 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:42.405773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:42.405773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:42.405830 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:42.406085 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:42.406086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:42.406086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:42.406212 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:42.406332 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.406655 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.406754 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:16:42.407038 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.396536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40405 (local address 127.16.79.1:56577)
0620 23:16:42.396783 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:42.397122 (+ 339us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:42.397843 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:42.398140 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:42.398145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:42.398917 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:42.399174 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:42.399178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.400025 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.400026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:42.401055 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:42.401058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.401518 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.401519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:42.401664 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.402314 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:42.402323 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:42.403714 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:42.405447 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:42.405454 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:42.405456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:42.405677 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:42.405901 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:42.405902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:42.405903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:42.406012 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:42.406338 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:42.406345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:42.406513 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 23:16:42.406624 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 23:16:42.406803 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.2@KRBTEST.COM" created.
I20260620 23:16:42.407544 3599 heartbeater.cc:347] Connected to a master server at 127.16.79.62:40405
I20260620 23:16:42.407648 3599 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:42.407835 3599 heartbeater.cc:511] Master 127.16.79.62:40405 requested a full tablet report, sending...
I20260620 23:16:42.408368 3407 ts_manager.cc:194] Registered new tserver with Master: 4f602e444b1c4cb281a43b40cc0391aa (127.16.79.1:41983)
I20260620 23:16:42.409087 3407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.1@KRBTEST.COM'} at 127.16.79.1:56577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
Entry for principal kudu/127.16.79.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab.
I20260620 23:16:42.424023 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:40405
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:42.505184 3604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:42.505492 3604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:42.505535 3604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:42.506964 3604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:42.507009 3604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:42.507037 3604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:16:42.508365 3604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.2.keytab
--principal=kudu/127.16.79.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:40405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:42.508536 3604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:42.508771 3604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:42.514202 3609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:42.514272 3610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:42.514585 3612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:42.514894 3604 server_base.cc:1034] running on GCE node
I20260620 23:16:42.515072 3604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:42.515257 3604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:42.516450 3604 hybrid_clock.cc:648] HybridClock initialized: now 1781997402516430 us; error 31 us; skew 500 ppm
Jun 20 23:16:42 dist-test-slave-pswb krb5kdc[3362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:42.519129 3604 init.cc:377] Logged in from keytab as kudu/127.16.79.2@KRBTEST.COM (short username kudu)
I20260620 23:16:42.519613 3604 webserver.cc:460] Webserver started at http://127.16.79.2:37981/ using document root <none> and password file <none>
I20260620 23:16:42.519845 3604 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:42.519910 3604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:42.520043 3604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:42.521143 3604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "301f486cab824caa91b3faf1c0d5a2ce"
format_stamp: "Formatted at 2026-06-20 23:16:42 on dist-test-slave-pswb"
server_key: "68281050536a2e10b0243579d566080a"
server_key_iv: "b177ad6759d6ebbec90dcfe5ad81fcc4"
server_key_version: "encryptionkey@0"
I20260620 23:16:42.521507 3604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "301f486cab824caa91b3faf1c0d5a2ce"
format_stamp: "Formatted at 2026-06-20 23:16:42 on dist-test-slave-pswb"
server_key: "68281050536a2e10b0243579d566080a"
server_key_iv: "b177ad6759d6ebbec90dcfe5ad81fcc4"
server_key_version: "encryptionkey@0"
I20260620 23:16:42.522903 3604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:16:42.523737 3619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:42.523953 3604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:42.524051 3604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "301f486cab824caa91b3faf1c0d5a2ce"
format_stamp: "Formatted at 2026-06-20 23:16:42 on dist-test-slave-pswb"
server_key: "68281050536a2e10b0243579d566080a"
server_key_iv: "b177ad6759d6ebbec90dcfe5ad81fcc4"
server_key_version: "encryptionkey@0"
I20260620 23:16:42.524134 3604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:42.536194 3604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:42.538584 3604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:42.538755 3604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:42.539008 3604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:42.539371 3604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:42.539429 3604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:42.539474 3604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:42.539503 3604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:42.546965 3604 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:32779
I20260620 23:16:42.547008 3732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:32779 every 8 connection(s)
I20260620 23:16:42.547302 3604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:16:42.549489 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 3604
I20260620 23:16:42.549623 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:16:42.549813 16700 external_mini_cluster.cc:1428] Setting key 42023a7a7940043a9a0e1f53ff4c2220
Jun 20 23:16:42 dist-test-slave-pswb krb5kdc[3362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.2@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.79.3@KRBTEST.COM; defaulting to no policy
I20260620 23:16:42.558118 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.547998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:34495 (local address 127.16.79.62:40405)
0620 23:16:42.548067 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.548069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.548299 (+ 230us) server_negotiation.cc:407] Connection header received
0620 23:16:42.549157 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.549157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.549167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.549224 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:42.550381 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.550937 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.552332 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.552664 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.552783 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.555126 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:42.555128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:42.555128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:42.555136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:42.556592 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:42.557069 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:42.557070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:42.557071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:42.557138 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:42.557415 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:42.557416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:42.557417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:42.557538 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:42.557640 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.557937 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.558023 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:16:42.558320 3735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.547963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40405 (local address 127.16.79.2:34495)
0620 23:16:42.548144 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:42.548189 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:42.549036 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:42.549313 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:42.549316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:42.550048 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:42.550295 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:42.550301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.551031 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.551032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:42.552240 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:42.552243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.552788 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.552789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:42.552906 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.553556 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:42.553564 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:42.555006 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:42.556724 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:42.556732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:42.556734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:42.556971 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:42.557222 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:42.557223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:42.557223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:42.557341 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:42.557633 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:42.557639 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:42.557799 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 23:16:42.557915 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 23:16:42.558072 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260620 23:16:42.558789 3733 heartbeater.cc:347] Connected to a master server at 127.16.79.62:40405
I20260620 23:16:42.558888 3733 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:42.559075 3733 heartbeater.cc:511] Master 127.16.79.62:40405 requested a full tablet report, sending...
I20260620 23:16:42.559489 3407 ts_manager.cc:194] Registered new tserver with Master: 301f486cab824caa91b3faf1c0d5a2ce (127.16.79.2:32779)
I20260620 23:16:42.559880 3407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.2@KRBTEST.COM'} at 127.16.79.2:34495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
Entry for principal kudu/127.16.79.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab.
I20260620 23:16:42.575567 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:40405
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:42.656117 3738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:42.656303 3738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:42.656325 3738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:42.657442 3738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:42.657485 3738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:42.657512 3738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:16:42.659085 3738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.3.keytab
--principal=kudu/127.16.79.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:40405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:42.659332 3738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:42.659564 3738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:42.664601 3743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:42.664585 3746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:42.664585 3744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:42.664826 3738 server_base.cc:1034] running on GCE node
I20260620 23:16:42.665114 3738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:42.665413 3738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:42.666622 3738 hybrid_clock.cc:648] HybridClock initialized: now 1781997402666603 us; error 38 us; skew 500 ppm
Jun 20 23:16:42 dist-test-slave-pswb krb5kdc[3362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:42.669414 3738 init.cc:377] Logged in from keytab as kudu/127.16.79.3@KRBTEST.COM (short username kudu)
I20260620 23:16:42.669889 3738 webserver.cc:460] Webserver started at http://127.16.79.3:34897/ using document root <none> and password file <none>
I20260620 23:16:42.670125 3738 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:42.670197 3738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:42.670347 3738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:42.671367 3738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "8a666e180e5d4783a18be4214bf9ffdb"
format_stamp: "Formatted at 2026-06-20 23:16:42 on dist-test-slave-pswb"
server_key: "48dfb292b3f1ebb80f23b9da8f35522e"
server_key_iv: "ae5eb2882d4a23fc249eddc190533802"
server_key_version: "encryptionkey@0"
I20260620 23:16:42.671717 3738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "8a666e180e5d4783a18be4214bf9ffdb"
format_stamp: "Formatted at 2026-06-20 23:16:42 on dist-test-slave-pswb"
server_key: "48dfb292b3f1ebb80f23b9da8f35522e"
server_key_iv: "ae5eb2882d4a23fc249eddc190533802"
server_key_version: "encryptionkey@0"
I20260620 23:16:42.673082 3738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:16:42.673800 3753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:42.673985 3738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:16:42.674098 3738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "8a666e180e5d4783a18be4214bf9ffdb"
format_stamp: "Formatted at 2026-06-20 23:16:42 on dist-test-slave-pswb"
server_key: "48dfb292b3f1ebb80f23b9da8f35522e"
server_key_iv: "ae5eb2882d4a23fc249eddc190533802"
server_key_version: "encryptionkey@0"
I20260620 23:16:42.674191 3738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:42.706816 3738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:42.708979 3738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:42.709087 3738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:42.709381 3738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:42.709694 3738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:42.709735 3738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:42.709757 3738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:42.709770 3738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:42.716884 3738 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:44021
I20260620 23:16:42.716926 3866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:44021 every 8 connection(s)
I20260620 23:16:42.717231 3738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:16:42.720959 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 3738
I20260620 23:16:42.721103 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:16:42.721338 16700 external_mini_cluster.cc:1428] Setting key 62f598b899dbc192250993f0a51f7804
Jun 20 23:16:42 dist-test-slave-pswb krb5kdc[3362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.3@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:16:42.727483 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.718042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:33017 (local address 127.16.79.62:40405)
0620 23:16:42.718104 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.718106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.718364 (+ 258us) server_negotiation.cc:407] Connection header received
0620 23:16:42.718891 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.718892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.718904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.718960 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:42.719858 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.720370 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.721553 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.721854 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.721925 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.724653 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:42.724654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:42.724654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:42.724660 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:42.725933 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:42.726485 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:42.726486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:42.726486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:42.726538 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:42.726865 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:42.726865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:42.726866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:42.726978 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:42.727041 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.727362 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.727420 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:16:42.727689 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.718050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40405 (local address 127.16.79.3:33017)
0620 23:16:42.718237 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:42.718274 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:42.718787 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:42.719032 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:42.719035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:42.719563 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:42.719755 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:42.719759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.720477 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.720477 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:42.721458 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:42.721462 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.722007 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.722009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:42.722170 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.722830 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:42.722843 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:42.724558 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:42.726086 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:42.726094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:42.726097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:42.726395 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:42.726621 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:42.726621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:42.726622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:42.726769 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:42.727081 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:42.727087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:42.727205 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 23:16:42.727301 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:16:42.727431 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260620 23:16:42.728338 3867 heartbeater.cc:347] Connected to a master server at 127.16.79.62:40405
I20260620 23:16:42.728442 3867 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:42.728610 3867 heartbeater.cc:511] Master 127.16.79.62:40405 requested a full tablet report, sending...
I20260620 23:16:42.729059 3407 ts_manager.cc:194] Registered new tserver with Master: 8a666e180e5d4783a18be4214bf9ffdb (127.16.79.3:44021)
I20260620 23:16:42.729517 3407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.79.3@KRBTEST.COM'} at 127.16.79.3:33017
I20260620 23:16:42.733737 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:16:42.741779 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.735216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50684 (local address 127.16.79.62:40405)
0620 23:16:42.735278 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.735280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.735288 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:16:42.735319 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.735320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.735327 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.735381 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:42.736109 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.736565 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.737416 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.737655 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.737755 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.739235 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:42.739237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:42.739237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:42.739245 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:42.740657 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:42.741087 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:42.741088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:42.741088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:42.741139 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:42.741332 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:42.741332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:42.741333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:42.741475 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:42.741554 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.741665 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.741722 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 23:16:42.743577 3407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:16:42.745292 3407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:16:42.752167 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.747855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:32779 (local address 127.0.0.1:36434)
0620 23:16:42.748011 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:42.748047 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:42.748112 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:42.748359 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:42.748362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:42.748364 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:42.748429 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:42.748432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.749838 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.749839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:42.750833 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:42.750836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.751738 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.751740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:42.751875 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.751896 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:16:42.751971 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:16:42.752055 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":97,"thread_start_us":55,"threads_started":1}
I20260620 23:16:42.752408 3882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.748049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:41983 (local address 127.0.0.1:60444)
0620 23:16:42.748415 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:42.748425 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:42.748477 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:42.748772 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:42.748773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:42.748775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:42.748818 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:42.748819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.749723 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.749725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:42.751108 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:42.751111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.752055 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.752056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:42.752114 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.752119 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:16:42.752180 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:16:42.752234 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":42,"thread_start_us":55,"threads_started":1}
I20260620 23:16:42.752477 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.747934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.16.79.3:44021)
0620 23:16:42.748258 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.748263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.748274 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:16:42.748307 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.748310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.748325 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.748393 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:42.748578 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.749722 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.751277 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.751871 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.751945 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.752005 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.752293 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.752373 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":82,"threads_started":1}
I20260620 23:16:42.752672 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.748252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.16.79.1:41983)
0620 23:16:42.748411 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.748415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.748465 (+ 50us) server_negotiation.cc:407] Connection header received
0620 23:16:42.748539 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.748540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.748554 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.748645 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:42.748912 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.749621 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.751205 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.751900 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.751969 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.752021 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.752555 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.752596 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260620 23:16:42.752677 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.747995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.16.79.2:32779)
0620 23:16:42.748173 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.748178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.748189 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:16:42.748230 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.748232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.748248 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.748507 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:42.748571 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.749737 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.750914 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.751537 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.752333 (+ 796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.752440 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.752495 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.752571 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260620 23:16:42.752832 3879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.747860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:44021 (local address 127.0.0.1:42756)
0620 23:16:42.748179 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:42.748191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:42.748242 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:42.748460 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:42.748460 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:42.748463 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:42.748504 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:42.748505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.749922 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.749923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:42.751204 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:42.751206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.752041 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.752042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:42.752115 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.752120 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:16:42.752179 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 23:16:42.752231 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":138,"thread_start_us":76,"threads_started":1}
I20260620 23:16:42.753451 3667 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 (DEFAULT_TABLE table=test-table [id=bf5a4052e67f4b26916fd76f2ccd4fc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:42.753476 3800 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 (DEFAULT_TABLE table=test-table [id=bf5a4052e67f4b26916fd76f2ccd4fc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:42.753794 3667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:42.753810 3800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:42.756345 3884 tablet_bootstrap.cc:492] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb: Bootstrap starting.
I20260620 23:16:42.756810 3885 tablet_bootstrap.cc:492] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce: Bootstrap starting.
I20260620 23:16:42.757066 3884 tablet_bootstrap.cc:654] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:42.757083 3533 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 (DEFAULT_TABLE table=test-table [id=bf5a4052e67f4b26916fd76f2ccd4fc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:42.757416 3885 tablet_bootstrap.cc:654] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:42.757443 3533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:42.757622 3884 log.cc:824] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:42.757987 3885 log.cc:824] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:42.758404 3884 tablet_bootstrap.cc:492] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb: No bootstrap required, opened a new log
I20260620 23:16:42.758476 3884 ts_tablet_manager.cc:1397] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:16:42.758684 3885 tablet_bootstrap.cc:492] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce: No bootstrap required, opened a new log
I20260620 23:16:42.758749 3885 ts_tablet_manager.cc:1397] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 23:16:42.760231 3888 tablet_bootstrap.cc:492] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa: Bootstrap starting.
I20260620 23:16:42.760445 3884 raft_consensus.cc:348] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e180e5d4783a18be4214bf9ffdb" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44021 } } peers { permanent_uuid: "4f602e444b1c4cb281a43b40cc0391aa" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 41983 } } peers { permanent_uuid: "301f486cab824caa91b3faf1c0d5a2ce" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 32779 } }
I20260620 23:16:42.760625 3884 raft_consensus.cc:374] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:42.760665 3884 raft_consensus.cc:729] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e180e5d4783a18be4214bf9ffdb, State: Initialized, Role: FOLLOWER
I20260620 23:16:42.760622 3885 raft_consensus.cc:348] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e180e5d4783a18be4214bf9ffdb" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44021 } } peers { permanent_uuid: "4f602e444b1c4cb281a43b40cc0391aa" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 41983 } } peers { permanent_uuid: "301f486cab824caa91b3faf1c0d5a2ce" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 32779 } }
I20260620 23:16:42.760802 3885 raft_consensus.cc:374] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:42.760836 3885 raft_consensus.cc:729] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 301f486cab824caa91b3faf1c0d5a2ce, State: Initialized, Role: FOLLOWER
I20260620 23:16:42.760843 3884 consensus_queue.cc:260] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e180e5d4783a18be4214bf9ffdb" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44021 } } peers { permanent_uuid: "4f602e444b1c4cb281a43b40cc0391aa" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 41983 } } peers { permanent_uuid: "301f486cab824caa91b3faf1c0d5a2ce" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 32779 } }
I20260620 23:16:42.760910 3888 tablet_bootstrap.cc:654] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:42.760928 3885 consensus_queue.cc:260] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e180e5d4783a18be4214bf9ffdb" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44021 } } peers { permanent_uuid: "4f602e444b1c4cb281a43b40cc0391aa" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 41983 } } peers { permanent_uuid: "301f486cab824caa91b3faf1c0d5a2ce" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 32779 } }
I20260620 23:16:42.761112 3884 ts_tablet_manager.cc:1428] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 23:16:42.761164 3885 ts_tablet_manager.cc:1428] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 23:16:42.761269 3733 heartbeater.cc:503] Master 127.16.79.62:40405 was elected leader, sending a full tablet report...
I20260620 23:16:42.761297 3867 heartbeater.cc:503] Master 127.16.79.62:40405 was elected leader, sending a full tablet report...
I20260620 23:16:42.761439 3888 log.cc:824] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:42.762254 3888 tablet_bootstrap.cc:492] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa: No bootstrap required, opened a new log
I20260620 23:16:42.762319 3888 ts_tablet_manager.cc:1397] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:16:42.763626 3888 raft_consensus.cc:348] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e180e5d4783a18be4214bf9ffdb" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44021 } } peers { permanent_uuid: "4f602e444b1c4cb281a43b40cc0391aa" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 41983 } } peers { permanent_uuid: "301f486cab824caa91b3faf1c0d5a2ce" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 32779 } }
I20260620 23:16:42.763839 3888 raft_consensus.cc:374] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:42.763876 3888 raft_consensus.cc:729] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f602e444b1c4cb281a43b40cc0391aa, State: Initialized, Role: FOLLOWER
I20260620 23:16:42.764005 3888 consensus_queue.cc:260] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e180e5d4783a18be4214bf9ffdb" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44021 } } peers { permanent_uuid: "4f602e444b1c4cb281a43b40cc0391aa" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 41983 } } peers { permanent_uuid: "301f486cab824caa91b3faf1c0d5a2ce" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 32779 } }
I20260620 23:16:42.764284 3888 ts_tablet_manager.cc:1428] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:16:42.764297 3599 heartbeater.cc:503] Master 127.16.79.62:40405 was elected leader, sending a full tablet report...
W20260620 23:16:42.797904 3734 tablet.cc:2369] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:16:42.896167 3600 tablet.cc:2369] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:16:42.967904 3868 tablet.cc:2369] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:16:43.026319 3890 raft_consensus.cc:482] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:16:43.026438 3890 raft_consensus.cc:504] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e180e5d4783a18be4214bf9ffdb" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44021 } } peers { permanent_uuid: "4f602e444b1c4cb281a43b40cc0391aa" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 41983 } } peers { permanent_uuid: "301f486cab824caa91b3faf1c0d5a2ce" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 32779 } }
I20260620 23:16:43.026788 3890 leader_election.cc:288] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e180e5d4783a18be4214bf9ffdb (127.16.79.3:44021), 4f602e444b1c4cb281a43b40cc0391aa (127.16.79.1:41983)
I20260620 23:16:43.030315 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.027090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:36849 (local address 127.16.79.3:44021)
0620 23:16:43.027142 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.027144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.027165 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:16:43.027217 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.027218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.027232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.027305 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:43.027500 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.028252 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.029312 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.029784 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.029839 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.029848 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.030125 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.030208 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:16:43.030296 3735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.026954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:44021 (local address 127.16.79.2:36849)
0620 23:16:43.027014 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:43.027027 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:43.027096 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:43.027323 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:43.027324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:43.027327 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:43.027423 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.027425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.028354 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.028355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.029165 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.029168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.029867 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.029868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.029950 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.029954 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:16:43.030024 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 23:16:43.030183 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":20}
I20260620 23:16:43.030565 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.027192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:46793 (local address 127.16.79.1:41983)
0620 23:16:43.027244 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.027246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.027275 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:16:43.027340 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.027341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.027353 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.027406 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:43.027568 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.028275 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.029522 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.029951 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.030144 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.030176 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.030393 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.030472 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:16:43.030674 3893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.027075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:41983 (local address 127.16.79.2:46793)
0620 23:16:43.027226 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:43.027235 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:43.027287 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:43.027428 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:43.027429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:43.027430 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:43.027495 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.027496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.028365 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.028366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.029420 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.029423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.030133 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.030136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.030257 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.030264 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:16:43.030381 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:16:43.030555 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260620 23:16:43.030776 3821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0a5ec2989e4f8f8b4d6b6b3864d8b7" candidate_uuid: "301f486cab824caa91b3faf1c0d5a2ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e180e5d4783a18be4214bf9ffdb" is_pre_election: true
I20260620 23:16:43.030969 3821 raft_consensus.cc:2454] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 301f486cab824caa91b3faf1c0d5a2ce in term 0.
I20260620 23:16:43.031152 3553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0a5ec2989e4f8f8b4d6b6b3864d8b7" candidate_uuid: "301f486cab824caa91b3faf1c0d5a2ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f602e444b1c4cb281a43b40cc0391aa" is_pre_election: true
I20260620 23:16:43.031229 3623 leader_election.cc:302] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 301f486cab824caa91b3faf1c0d5a2ce, 8a666e180e5d4783a18be4214bf9ffdb; no voters:
I20260620 23:16:43.031352 3553 raft_consensus.cc:2454] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 301f486cab824caa91b3faf1c0d5a2ce in term 0.
I20260620 23:16:43.031463 3890 raft_consensus.cc:2781] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:16:43.031522 3890 raft_consensus.cc:482] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:16:43.031543 3890 raft_consensus.cc:3037] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:43.032344 3890 raft_consensus.cc:504] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e180e5d4783a18be4214bf9ffdb" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44021 } } peers { permanent_uuid: "4f602e444b1c4cb281a43b40cc0391aa" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 41983 } } peers { permanent_uuid: "301f486cab824caa91b3faf1c0d5a2ce" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 32779 } }
I20260620 23:16:43.032512 3890 leader_election.cc:288] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e180e5d4783a18be4214bf9ffdb (127.16.79.3:44021), 4f602e444b1c4cb281a43b40cc0391aa (127.16.79.1:41983)
I20260620 23:16:43.032749 3821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0a5ec2989e4f8f8b4d6b6b3864d8b7" candidate_uuid: "301f486cab824caa91b3faf1c0d5a2ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e180e5d4783a18be4214bf9ffdb"
I20260620 23:16:43.032858 3821 raft_consensus.cc:3037] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:43.032838 3553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0a5ec2989e4f8f8b4d6b6b3864d8b7" candidate_uuid: "301f486cab824caa91b3faf1c0d5a2ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f602e444b1c4cb281a43b40cc0391aa"
I20260620 23:16:43.032946 3553 raft_consensus.cc:3037] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:43.033767 3821 raft_consensus.cc:2454] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 301f486cab824caa91b3faf1c0d5a2ce in term 1.
I20260620 23:16:43.033778 3553 raft_consensus.cc:2454] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 301f486cab824caa91b3faf1c0d5a2ce in term 1.
I20260620 23:16:43.033952 3623 leader_election.cc:302] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 301f486cab824caa91b3faf1c0d5a2ce, 4f602e444b1c4cb281a43b40cc0391aa; no voters:
I20260620 23:16:43.034092 3890 raft_consensus.cc:2781] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:43.034273 3890 raft_consensus.cc:686] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 1 LEADER]: Becoming Leader. State: Replica: 301f486cab824caa91b3faf1c0d5a2ce, State: Running, Role: LEADER
I20260620 23:16:43.034379 3890 consensus_queue.cc:237] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e180e5d4783a18be4214bf9ffdb" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44021 } } peers { permanent_uuid: "4f602e444b1c4cb281a43b40cc0391aa" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 41983 } } peers { permanent_uuid: "301f486cab824caa91b3faf1c0d5a2ce" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 32779 } }
I20260620 23:16:43.035312 3406 catalog_manager.cc:5495] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 301f486cab824caa91b3faf1c0d5a2ce (127.16.79.2). New cstate: current_term: 1 leader_uuid: "301f486cab824caa91b3faf1c0d5a2ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e180e5d4783a18be4214bf9ffdb" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f602e444b1c4cb281a43b40cc0391aa" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 41983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "301f486cab824caa91b3faf1c0d5a2ce" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 32779 } health_report { overall_health: HEALTHY } } }
I20260620 23:16:43.107735 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.104918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.16.79.2:32779)
0620 23:16:43.105072 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.105074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.105086 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:16:43.105136 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.105137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.105164 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.105245 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:43.105481 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.106074 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.106937 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.107156 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.107226 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.107378 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:43.107488 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:43.107551 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.107585 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.107602 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 23:16:43.109668 3821 raft_consensus.cc:1261] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb [term 1 FOLLOWER]: Refusing update from remote peer 301f486cab824caa91b3faf1c0d5a2ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:43.109668 3553 raft_consensus.cc:1261] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 4f602e444b1c4cb281a43b40cc0391aa [term 1 FOLLOWER]: Refusing update from remote peer 301f486cab824caa91b3faf1c0d5a2ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:43.110139 3894 consensus_queue.cc:1035] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f602e444b1c4cb281a43b40cc0391aa" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 41983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:43.110285 3890 consensus_queue.cc:1035] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e180e5d4783a18be4214bf9ffdb" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 44021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:43.112799 3898 mvcc.cc:205] Tried to move back new op lower bound from 7299061363134222336 to 7299061362829348864. Current Snapshot: MvccSnapshot[applied={T|T < 7299061363134222336}]
I20260620 23:16:43.114806 3407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:16:43.114913 3407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:16:43.115731 3407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 355fbeb71bf343aa9921facaa434df96: Sending DeleteTablet for 3 replicas of tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7
I20260620 23:16:43.116046 3667 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:43 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:36434
I20260620 23:16:43.116101 3533 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:43 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:60444
I20260620 23:16:43.116122 3800 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:43 UTC) from {username='kudu', principal='kudu/127.16.79.62@KRBTEST.COM'} at 127.0.0.1:42756
I20260620 23:16:43.116319 3909 tablet_replica.cc:330] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb: stopping tablet replica
I20260620 23:16:43.116513 3909 raft_consensus.cc:2229] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:16:43.116614 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 3470
I20260620 23:16:43.117455 3909 raft_consensus.cc:2258] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:43.117947 3909 ts_tablet_manager.cc:1905] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:16:43.116524 3907 tablet_replica.cc:330] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce: stopping tablet replica
I20260620 23:16:43.118275 3907 raft_consensus.cc:2229] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:16:43.118424 3907 raft_consensus.cc:2258] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:43.118880 3907 ts_tablet_manager.cc:1905] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:16:43.119678 3909 ts_tablet_manager.cc:1918] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:16:43.119766 3909 log.cc:1196] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/6d0a5ec2989e4f8f8b4d6b6b3864d8b7
I20260620 23:16:43.120048 3909 ts_tablet_manager.cc:1939] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 8a666e180e5d4783a18be4214bf9ffdb: Deleting consensus metadata
I20260620 23:16:43.120457 3394 catalog_manager.cc:4841] TS 8a666e180e5d4783a18be4214bf9ffdb (127.16.79.3:44021): tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 (table test-table [id=bf5a4052e67f4b26916fd76f2ccd4fc6]) successfully deleted
I20260620 23:16:43.120751 3907 ts_tablet_manager.cc:1918] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:16:43.120819 3907 log.cc:1196] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/6d0a5ec2989e4f8f8b4d6b6b3864d8b7
I20260620 23:16:43.121042 3907 ts_tablet_manager.cc:1939] T 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 P 301f486cab824caa91b3faf1c0d5a2ce: Deleting consensus metadata
I20260620 23:16:43.121415 3392 catalog_manager.cc:4841] TS 301f486cab824caa91b3faf1c0d5a2ce (127.16.79.2:32779): tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7 (table test-table [id=bf5a4052e67f4b26916fd76f2ccd4fc6]) successfully deleted
W20260620 23:16:43.123754 3394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:41983 (error 108)
I20260620 23:16:43.123961 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 3604
I20260620 23:16:43.126443 3879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.126296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:41983 (local address 127.0.0.1:60460)
0620 23:16:43.126363 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:43.126403 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:41983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 23:16:43.126667 3394 catalog_manager.cc:4584] TS 4f602e444b1c4cb281a43b40cc0391aa (127.16.79.1:41983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d0a5ec2989e4f8f8b4d6b6b3864d8b7: Network error: Client connection negotiation failed: client connection to 127.16.79.1:41983: connect: Connection refused (error 111)
I20260620 23:16:43.129773 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 3738
I20260620 23:16:43.134294 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 3375
2026-06-20T23:16:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2238 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T23:16:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:43Z Disabled control of system clock
I20260620 23:16:43.163574 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36751
--webserver_interface=127.16.79.62
--webserver_port=0
--builtin_ntp_servers=127.16.79.20:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:36751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:16:43.242988 3914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:43.243218 3914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:43.243243 3914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:43.244450 3914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:43.244490 3914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:43.244504 3914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:43.244517 3914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:43.244575 3914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:43.245918 3914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:36751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:43.246168 3914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:43.246466 3914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:43.251121 3919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:43.251134 3920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:43.251139 3922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:43.251502 3914 server_base.cc:1034] running on GCE node
I20260620 23:16:43.251742 3914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:43.252023 3914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:43.253171 3914 hybrid_clock.cc:648] HybridClock initialized: now 1781997403253158 us; error 27 us; skew 500 ppm
I20260620 23:16:43.254408 3914 webserver.cc:460] Webserver started at http://127.16.79.62:39555/ using document root <none> and password file <none>
I20260620 23:16:43.254647 3914 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:43.254693 3914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:43.254840 3914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:43.256008 3914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "beaea6cbcfba48ddbb23c6f8b4495917"
format_stamp: "Formatted at 2026-06-20 23:16:43 on dist-test-slave-pswb"
server_key: "0dcd9c8061349790ff02967d7a9757a1"
server_key_iv: "1a703d5e26fb0bd79ed819a6ebaa4e7d"
server_key_version: "encryptionkey@0"
I20260620 23:16:43.256444 3914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "beaea6cbcfba48ddbb23c6f8b4495917"
format_stamp: "Formatted at 2026-06-20 23:16:43 on dist-test-slave-pswb"
server_key: "0dcd9c8061349790ff02967d7a9757a1"
server_key_iv: "1a703d5e26fb0bd79ed819a6ebaa4e7d"
server_key_version: "encryptionkey@0"
I20260620 23:16:43.257854 3914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:16:43.258744 3928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:43.258994 3914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:43.259102 3914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "beaea6cbcfba48ddbb23c6f8b4495917"
format_stamp: "Formatted at 2026-06-20 23:16:43 on dist-test-slave-pswb"
server_key: "0dcd9c8061349790ff02967d7a9757a1"
server_key_iv: "1a703d5e26fb0bd79ed819a6ebaa4e7d"
server_key_version: "encryptionkey@0"
I20260620 23:16:43.259194 3914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:43.275209 3914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:43.275506 3914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:43.275852 3914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:43.281289 3914 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:36751
I20260620 23:16:43.281337 3980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:36751 every 8 connection(s)
I20260620 23:16:43.281726 3914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:43.282946 3981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:43.285463 3981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917: Bootstrap starting.
I20260620 23:16:43.286080 3981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:43.286408 3981 log.cc:824] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:43.287135 3981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917: No bootstrap required, opened a new log
I20260620 23:16:43.288164 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 3914
I20260620 23:16:43.288267 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:43.288504 16700 external_mini_cluster.cc:1428] Setting key 27e7b6aa4b1ebdbad528bc5750bd7d8b
I20260620 23:16:43.289594 3981 raft_consensus.cc:348] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beaea6cbcfba48ddbb23c6f8b4495917" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36751 } }
I20260620 23:16:43.289785 3981 raft_consensus.cc:374] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:43.289832 3981 raft_consensus.cc:729] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beaea6cbcfba48ddbb23c6f8b4495917, State: Initialized, Role: FOLLOWER
I20260620 23:16:43.289968 3981 consensus_queue.cc:260] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beaea6cbcfba48ddbb23c6f8b4495917" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36751 } }
I20260620 23:16:43.290051 3981 raft_consensus.cc:388] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:43.290074 3981 raft_consensus.cc:482] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:43.290153 3981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:43.290735 3981 raft_consensus.cc:504] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beaea6cbcfba48ddbb23c6f8b4495917" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36751 } }
I20260620 23:16:43.290824 3981 leader_election.cc:302] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beaea6cbcfba48ddbb23c6f8b4495917; no voters:
I20260620 23:16:43.291007 3981 leader_election.cc:288] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:43.291061 3986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:43.291184 3986 raft_consensus.cc:686] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [term 1 LEADER]: Becoming Leader. State: Replica: beaea6cbcfba48ddbb23c6f8b4495917, State: Running, Role: LEADER
I20260620 23:16:43.291262 3986 consensus_queue.cc:237] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beaea6cbcfba48ddbb23c6f8b4495917" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36751 } }
I20260620 23:16:43.291288 3981 sys_catalog.cc:564] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:43.291563 3986 sys_catalog.cc:455] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [sys.catalog]: SysCatalogTable state changed. Reason: New leader beaea6cbcfba48ddbb23c6f8b4495917. Latest consensus state: current_term: 1 leader_uuid: "beaea6cbcfba48ddbb23c6f8b4495917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beaea6cbcfba48ddbb23c6f8b4495917" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36751 } } }
I20260620 23:16:43.291625 3986 sys_catalog.cc:458] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:43.291576 3987 sys_catalog.cc:455] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beaea6cbcfba48ddbb23c6f8b4495917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beaea6cbcfba48ddbb23c6f8b4495917" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36751 } } }
I20260620 23:16:43.291769 3987 sys_catalog.cc:458] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:43.291930 3990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:43.292512 3990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:43.295181 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.289207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.16.79.62:36751)
0620 23:16:43.289448 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.289460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.289496 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:16:43.289618 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.289626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.289699 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.289854 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:43.290169 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.291792 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.292938 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.293524 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.293962 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.294386 (+ 424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:43.294389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:43.294389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:43.294583 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:43.294620 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:43.294634 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:43.294871 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.294903 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.294979 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
I20260620 23:16:43.295367 3990 catalog_manager.cc:1347] Generated new cluster ID: 929c20c73a1141f3892224815432ec41
I20260620 23:16:43.295449 3990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:43.308583 3990 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:43.309069 3990 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:43.318229 3990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917: Generated new TSK 0
I20260620 23:16:43.318413 3990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:43.320389 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.1:0
--local_ip_for_outbound_sockets=127.16.79.1
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:36751
--builtin_ntp_servers=127.16.79.20:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:16:43.411092 4005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:43.411372 4005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:43.411412 4005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:43.413220 4005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:43.413277 4005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:43.413318 4005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.1
I20260620 23:16:43.415583 4005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.79.1
--webserver_port=0
--tserver_master_addrs=127.16.79.62:36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.1
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:43.415861 4005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:43.416235 4005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:43.422569 4011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:43.422570 4010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:43.422564 4013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:43.424894 4005 server_base.cc:1034] running on GCE node
I20260620 23:16:43.425189 4005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:43.425491 4005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:43.426622 4005 hybrid_clock.cc:648] HybridClock initialized: now 1781997403426600 us; error 46 us; skew 500 ppm
I20260620 23:16:43.428076 4005 webserver.cc:460] Webserver started at http://127.16.79.1:45241/ using document root <none> and password file <none>
I20260620 23:16:43.428347 4005 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:43.428418 4005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:43.428550 4005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:43.429715 4005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/data/instance:
uuid: "3286108e74b642298460f1b4dcbbe2b2"
format_stamp: "Formatted at 2026-06-20 23:16:43 on dist-test-slave-pswb"
server_key: "4c7b68bfa6a8749f72de2fb8cd4c4c51"
server_key_iv: "5d2e26e232e1541ee5d0f3f619ecb45d"
server_key_version: "encryptionkey@0"
I20260620 23:16:43.430066 4005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance:
uuid: "3286108e74b642298460f1b4dcbbe2b2"
format_stamp: "Formatted at 2026-06-20 23:16:43 on dist-test-slave-pswb"
server_key: "4c7b68bfa6a8749f72de2fb8cd4c4c51"
server_key_iv: "5d2e26e232e1541ee5d0f3f619ecb45d"
server_key_version: "encryptionkey@0"
I20260620 23:16:43.431334 4005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:16:43.432202 4019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:43.432516 4005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:43.432619 4005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/wal
uuid: "3286108e74b642298460f1b4dcbbe2b2"
format_stamp: "Formatted at 2026-06-20 23:16:43 on dist-test-slave-pswb"
server_key: "4c7b68bfa6a8749f72de2fb8cd4c4c51"
server_key_iv: "5d2e26e232e1541ee5d0f3f619ecb45d"
server_key_version: "encryptionkey@0"
I20260620 23:16:43.432727 4005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:43.452472 4005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:43.452797 4005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:43.452948 4005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:43.453169 4005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:43.453492 4005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:43.453547 4005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:43.453586 4005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:43.453615 4005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:43.459378 4005 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.1:36269
I20260620 23:16:43.459405 4132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.1:36269 every 8 connection(s)
I20260620 23:16:43.459723 4005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/data/info.pb
I20260620 23:16:43.465364 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.460443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:37707 (local address 127.16.79.62:36751)
0620 23:16:43.460526 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.460529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.460745 (+ 216us) server_negotiation.cc:407] Connection header received
0620 23:16:43.461344 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.461345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.461358 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.461412 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:43.462260 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.463115 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.464018 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.464197 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.464250 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.465064 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:43.465065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:43.465065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:43.465069 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:43.465083 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:43.465087 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:43.465147 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.465282 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.465304 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:16:43.465643 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.460407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:36751 (local address 127.16.79.1:37707)
0620 23:16:43.460602 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:43.460635 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:43.461177 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:43.461484 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:43.461487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:43.461495 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:43.462123 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.462129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.463196 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.463197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.463945 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.463946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.464274 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.464275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.464372 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.464898 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:16:43.464899 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:43.464925 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:16:43.464944 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:43.465186 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:43.465193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:43.465196 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:43.465271 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:16:43.465361 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260620 23:16:43.466264 4133 heartbeater.cc:347] Connected to a master server at 127.16.79.62:36751
I20260620 23:16:43.466377 4133 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:43.466563 4133 heartbeater.cc:511] Master 127.16.79.62:36751 requested a full tablet report, sending...
I20260620 23:16:43.467073 3945 ts_manager.cc:194] Registered new tserver with Master: 3286108e74b642298460f1b4dcbbe2b2 (127.16.79.1:36269)
I20260620 23:16:43.467681 3945 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.79.1:37707
I20260620 23:16:43.469597 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 4005
I20260620 23:16:43.469753 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-0/wal/instance
I20260620 23:16:43.469945 16700 external_mini_cluster.cc:1428] Setting key 665142958c825eb558f40592e766667b
I20260620 23:16:43.470849 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.2:0
--local_ip_for_outbound_sockets=127.16.79.2
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:36751
--builtin_ntp_servers=127.16.79.20:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:16:43.558930 4136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:43.559131 4136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:43.559154 4136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:43.560333 4136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:43.560371 4136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:43.560396 4136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.2
I20260620 23:16:43.561868 4136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.79.2
--webserver_port=0
--tserver_master_addrs=127.16.79.62:36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.2
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:43.562130 4136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:43.562429 4136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:43.567433 4144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:43.567445 4141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:43.567442 4142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:43.567703 4136 server_base.cc:1034] running on GCE node
I20260620 23:16:43.567972 4136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:43.568226 4136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:43.569402 4136 hybrid_clock.cc:648] HybridClock initialized: now 1781997403569447 us; error 88 us; skew 500 ppm
I20260620 23:16:43.570776 4136 webserver.cc:460] Webserver started at http://127.16.79.2:45687/ using document root <none> and password file <none>
I20260620 23:16:43.571041 4136 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:43.571111 4136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:43.571246 4136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:43.572232 4136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/data/instance:
uuid: "7acd38c33ba240c5a9ff2b0297c10cac"
format_stamp: "Formatted at 2026-06-20 23:16:43 on dist-test-slave-pswb"
server_key: "a97fcd9c3911d31ae575c049b7c145d1"
server_key_iv: "26a9e4078a40c877c8085de9ea22c4c4"
server_key_version: "encryptionkey@0"
I20260620 23:16:43.572582 4136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance:
uuid: "7acd38c33ba240c5a9ff2b0297c10cac"
format_stamp: "Formatted at 2026-06-20 23:16:43 on dist-test-slave-pswb"
server_key: "a97fcd9c3911d31ae575c049b7c145d1"
server_key_iv: "26a9e4078a40c877c8085de9ea22c4c4"
server_key_version: "encryptionkey@0"
I20260620 23:16:43.573917 4136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:16:43.574572 4150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:43.574743 4136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:16:43.574851 4136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/wal
uuid: "7acd38c33ba240c5a9ff2b0297c10cac"
format_stamp: "Formatted at 2026-06-20 23:16:43 on dist-test-slave-pswb"
server_key: "a97fcd9c3911d31ae575c049b7c145d1"
server_key_iv: "26a9e4078a40c877c8085de9ea22c4c4"
server_key_version: "encryptionkey@0"
I20260620 23:16:43.574920 4136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:43.593782 4136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:43.594043 4136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:43.594192 4136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:43.594406 4136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:43.594714 4136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:43.594764 4136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:43.594805 4136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:43.594833 4136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:43.600472 4136 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.2:43681
I20260620 23:16:43.600507 4263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.2:43681 every 8 connection(s)
I20260620 23:16:43.600829 4136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/data/info.pb
I20260620 23:16:43.606086 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 4136
I20260620 23:16:43.606200 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/wal/instance
I20260620 23:16:43.606417 16700 external_mini_cluster.cc:1428] Setting key 8355e7b6133bf930cf5fea639deb6ffb
I20260620 23:16:43.607311 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.601779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.2:41675 (local address 127.16.79.62:36751)
0620 23:16:43.601857 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.601859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.602062 (+ 203us) server_negotiation.cc:407] Connection header received
0620 23:16:43.602510 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.602510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.602520 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.602579 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:43.603192 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.603717 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.605414 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.605637 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.605727 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.606992 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:43.606993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:43.606993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:43.606998 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:43.607016 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:43.607021 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:43.607081 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.607188 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.607217 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:16:43.607527 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.79.3:0
--local_ip_for_outbound_sockets=127.16.79.3
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:36751
--builtin_ntp_servers=127.16.79.20:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:16:43.607547 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.601735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:36751 (local address 127.16.79.2:41675)
0620 23:16:43.601938 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:43.601978 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:43.602431 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:43.602627 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:43.602630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:43.602638 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:43.603109 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.603112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.603901 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.603903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.605289 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.605294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.605816 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.605818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.606045 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.606859 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:16:43.606859 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:43.606886 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:16:43.606907 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:43.607092 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:43.607098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:43.607101 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:43.607173 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 23:16:43.607297 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260620 23:16:43.608040 4264 heartbeater.cc:347] Connected to a master server at 127.16.79.62:36751
I20260620 23:16:43.608134 4264 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:43.608332 4264 heartbeater.cc:511] Master 127.16.79.62:36751 requested a full tablet report, sending...
I20260620 23:16:43.608759 3945 ts_manager.cc:194] Registered new tserver with Master: 7acd38c33ba240c5a9ff2b0297c10cac (127.16.79.2:43681)
I20260620 23:16:43.609167 3945 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.79.2:41675
W20260620 23:16:43.689759 4267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:43.689949 4267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:43.689970 4267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:43.691087 4267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:43.691123 4267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:43.691161 4267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.79.3
I20260620 23:16:43.692605 4267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.79.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.79.3
--webserver_port=0
--tserver_master_addrs=127.16.79.62:36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.79.3
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:43.692870 4267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:43.693194 4267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:43.698105 4273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:43.698156 4272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:43.698127 4275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:43.698282 4267 server_base.cc:1034] running on GCE node
I20260620 23:16:43.698554 4267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:43.698799 4267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:43.699946 4267 hybrid_clock.cc:648] HybridClock initialized: now 1781997403699939 us; error 32 us; skew 500 ppm
I20260620 23:16:43.701164 4267 webserver.cc:460] Webserver started at http://127.16.79.3:33343/ using document root <none> and password file <none>
I20260620 23:16:43.701388 4267 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:43.701434 4267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:43.701572 4267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:43.702581 4267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/data/instance:
uuid: "770d5930d28448de9835853a4a24623c"
format_stamp: "Formatted at 2026-06-20 23:16:43 on dist-test-slave-pswb"
server_key: "554a28cfef98d194d8d3aaa4de74b6af"
server_key_iv: "3e55311119adc0e4b4490ab2b21f9369"
server_key_version: "encryptionkey@0"
I20260620 23:16:43.702950 4267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance:
uuid: "770d5930d28448de9835853a4a24623c"
format_stamp: "Formatted at 2026-06-20 23:16:43 on dist-test-slave-pswb"
server_key: "554a28cfef98d194d8d3aaa4de74b6af"
server_key_iv: "3e55311119adc0e4b4490ab2b21f9369"
server_key_version: "encryptionkey@0"
I20260620 23:16:43.704166 4267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:16:43.704960 4281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:43.705130 4267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:43.705196 4267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/wal
uuid: "770d5930d28448de9835853a4a24623c"
format_stamp: "Formatted at 2026-06-20 23:16:43 on dist-test-slave-pswb"
server_key: "554a28cfef98d194d8d3aaa4de74b6af"
server_key_iv: "3e55311119adc0e4b4490ab2b21f9369"
server_key_version: "encryptionkey@0"
I20260620 23:16:43.705286 4267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:43.732950 4267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:43.733273 4267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:43.733419 4267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:43.733652 4267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:16:43.733996 4267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:16:43.734054 4267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:43.734097 4267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:16:43.734124 4267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:43.739912 4267 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.3:38345
I20260620 23:16:43.739954 4394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.3:38345 every 8 connection(s)
I20260620 23:16:43.740272 4267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/data/info.pb
I20260620 23:16:43.742746 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 4267
I20260620 23:16:43.742834 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/wal/instance
I20260620 23:16:43.743041 16700 external_mini_cluster.cc:1428] Setting key 7f6002e5c5b2fbbef2f9808ef45e9c85
I20260620 23:16:43.745586 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.740985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.3:38149 (local address 127.16.79.62:36751)
0620 23:16:43.741061 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.741063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.741298 (+ 235us) server_negotiation.cc:407] Connection header received
0620 23:16:43.741775 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.741776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.741786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.741835 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:43.742480 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.743082 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.744327 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.744513 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.744568 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.745395 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:43.745396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:43.745397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:43.745400 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:43.745413 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:43.745416 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:43.745460 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.745509 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.745530 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:16:43.745819 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.740953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:36751 (local address 127.16.79.3:38149)
0620 23:16:43.741181 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:43.741216 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:43.741694 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:43.741895 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:43.741898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:43.741906 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:43.742405 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.742409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.743214 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.743215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.744257 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.744261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.744586 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.744587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.744701 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.745263 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:16:43.745264 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:43.745291 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:16:43.745313 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:43.745461 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:43.745463 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:43.745465 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:16:43.745509 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 23:16:43.745579 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":108,"threads_started":1}
I20260620 23:16:43.746301 4395 heartbeater.cc:347] Connected to a master server at 127.16.79.62:36751
I20260620 23:16:43.746405 4395 heartbeater.cc:464] Registering TS with master...
I20260620 23:16:43.746573 4395 heartbeater.cc:511] Master 127.16.79.62:36751 requested a full tablet report, sending...
I20260620 23:16:43.746905 3945 ts_manager.cc:194] Registered new tserver with Master: 770d5930d28448de9835853a4a24623c (127.16.79.3:38345)
I20260620 23:16:43.747210 3945 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.79.3:38149
I20260620 23:16:43.754490 16700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:16:43.758872 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.755837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.16.79.62:36751)
0620 23:16:43.755883 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.755884 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.755980 (+ 96us) server_negotiation.cc:407] Connection header received
0620 23:16:43.756040 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.756041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.756049 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.756087 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:43.756228 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.756687 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.757554 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.757727 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.757784 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.758504 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:43.758506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:43.758506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:43.758513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:43.758569 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:43.758577 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:43.758680 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.758759 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.758796 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:16:43.760519 3945 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:16:43.762264 3945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:16:43.768572 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.764329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:36269 (local address 127.0.0.1:39388)
0620 23:16:43.764492 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:43.764526 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:43.764590 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:43.765181 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:43.765182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:43.765184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:43.765229 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.765230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.766391 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.766392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.767449 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.767451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.768289 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.768290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.768371 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.768384 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:16:43.768446 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:16:43.768498 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
I20260620 23:16:43.768587 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.764415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.16.79.1:36269)
0620 23:16:43.765013 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.765017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.765049 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:16:43.765085 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.765087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.765104 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.765206 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:43.765285 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.766308 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.767530 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.768201 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.768289 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.768377 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.768470 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.768506 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":88,"threads_started":1}
I20260620 23:16:43.768741 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.764387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:38345 (local address 127.0.0.1:60782)
0620 23:16:43.764572 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:43.764582 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:43.764623 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:43.764926 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:43.764928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:43.764930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:43.764995 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.764997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.766230 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.766230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.767641 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.767643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.768488 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.768489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.768546 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.768551 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:16:43.768602 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 23:16:43.768640 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260620 23:16:43.769047 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.764558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.16.79.3:38345)
0620 23:16:43.764740 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.764744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.764756 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:16:43.764793 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.764794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.764810 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.764912 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:43.765071 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.766154 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.767725 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.768304 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.768381 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.768453 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.768840 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.768891 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260620 23:16:43.769140 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.764392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:43681 (local address 127.0.0.1:58740)
0620 23:16:43.764713 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:43.764721 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:43.764762 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:43.764925 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:43.764928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:43.764932 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:43.764995 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.764998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.767058 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.767060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.767936 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.767939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.768886 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.768888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.768968 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.768971 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:43.769021 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 23:16:43.769058 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":9,"thread_start_us":57,"threads_started":1}
I20260620 23:16:43.769341 4408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.764560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.16.79.2:43681)
0620 23:16:43.764751 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.764755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.764765 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:16:43.764811 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.764813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.764826 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.764912 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:43.766254 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.766965 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.768025 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.768755 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.768824 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.768939 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.769135 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.769220 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260620 23:16:43.769677 4067 tablet_service.cc:1467] Processing CreateTablet for tablet 969930bf84274399a76f6c62d6f4c43f (DEFAULT_TABLE table=test-table [id=7a9d06098dc8417e899a90deb5ddda16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:43.769912 4329 tablet_service.cc:1467] Processing CreateTablet for tablet 969930bf84274399a76f6c62d6f4c43f (DEFAULT_TABLE table=test-table [id=7a9d06098dc8417e899a90deb5ddda16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:43.770033 4067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969930bf84274399a76f6c62d6f4c43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:43.770195 4329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969930bf84274399a76f6c62d6f4c43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:43.770390 4198 tablet_service.cc:1467] Processing CreateTablet for tablet 969930bf84274399a76f6c62d6f4c43f (DEFAULT_TABLE table=test-table [id=7a9d06098dc8417e899a90deb5ddda16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:16:43.770713 4198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969930bf84274399a76f6c62d6f4c43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:43.773386 4409 tablet_bootstrap.cc:492] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2: Bootstrap starting.
I20260620 23:16:43.774015 4409 tablet_bootstrap.cc:654] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:43.774350 4409 log.cc:824] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:43.774680 4410 tablet_bootstrap.cc:492] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c: Bootstrap starting.
I20260620 23:16:43.775018 4411 tablet_bootstrap.cc:492] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac: Bootstrap starting.
I20260620 23:16:43.775213 4409 tablet_bootstrap.cc:492] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2: No bootstrap required, opened a new log
I20260620 23:16:43.775247 4410 tablet_bootstrap.cc:654] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:43.775285 4409 ts_tablet_manager.cc:1397] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:16:43.775529 4410 log.cc:824] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:43.775643 4411 tablet_bootstrap.cc:654] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:43.775997 4411 log.cc:824] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:43.776201 4410 tablet_bootstrap.cc:492] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c: No bootstrap required, opened a new log
I20260620 23:16:43.776279 4410 ts_tablet_manager.cc:1397] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:16:43.777137 4411 tablet_bootstrap.cc:492] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac: No bootstrap required, opened a new log
I20260620 23:16:43.777209 4411 ts_tablet_manager.cc:1397] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:16:43.778175 4409 raft_consensus.cc:348] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3286108e74b642298460f1b4dcbbe2b2" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36269 } } peers { permanent_uuid: "7acd38c33ba240c5a9ff2b0297c10cac" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43681 } } peers { permanent_uuid: "770d5930d28448de9835853a4a24623c" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38345 } }
I20260620 23:16:43.778350 4409 raft_consensus.cc:374] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:43.778386 4409 raft_consensus.cc:729] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3286108e74b642298460f1b4dcbbe2b2, State: Initialized, Role: FOLLOWER
I20260620 23:16:43.778496 4409 consensus_queue.cc:260] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3286108e74b642298460f1b4dcbbe2b2" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36269 } } peers { permanent_uuid: "7acd38c33ba240c5a9ff2b0297c10cac" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43681 } } peers { permanent_uuid: "770d5930d28448de9835853a4a24623c" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38345 } }
I20260620 23:16:43.778658 4410 raft_consensus.cc:348] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3286108e74b642298460f1b4dcbbe2b2" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36269 } } peers { permanent_uuid: "7acd38c33ba240c5a9ff2b0297c10cac" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43681 } } peers { permanent_uuid: "770d5930d28448de9835853a4a24623c" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38345 } }
I20260620 23:16:43.778784 4409 ts_tablet_manager.cc:1428] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 23:16:43.778818 4410 raft_consensus.cc:374] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:43.778862 4410 raft_consensus.cc:729] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770d5930d28448de9835853a4a24623c, State: Initialized, Role: FOLLOWER
I20260620 23:16:43.778892 4133 heartbeater.cc:503] Master 127.16.79.62:36751 was elected leader, sending a full tablet report...
I20260620 23:16:43.778970 4410 consensus_queue.cc:260] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3286108e74b642298460f1b4dcbbe2b2" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36269 } } peers { permanent_uuid: "7acd38c33ba240c5a9ff2b0297c10cac" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43681 } } peers { permanent_uuid: "770d5930d28448de9835853a4a24623c" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38345 } }
I20260620 23:16:43.779232 4410 ts_tablet_manager.cc:1428] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:16:43.779250 4395 heartbeater.cc:503] Master 127.16.79.62:36751 was elected leader, sending a full tablet report...
I20260620 23:16:43.780350 4411 raft_consensus.cc:348] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3286108e74b642298460f1b4dcbbe2b2" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36269 } } peers { permanent_uuid: "7acd38c33ba240c5a9ff2b0297c10cac" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43681 } } peers { permanent_uuid: "770d5930d28448de9835853a4a24623c" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38345 } }
I20260620 23:16:43.780459 4411 raft_consensus.cc:374] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:43.780478 4411 raft_consensus.cc:729] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acd38c33ba240c5a9ff2b0297c10cac, State: Initialized, Role: FOLLOWER
I20260620 23:16:43.780591 4411 consensus_queue.cc:260] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3286108e74b642298460f1b4dcbbe2b2" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36269 } } peers { permanent_uuid: "7acd38c33ba240c5a9ff2b0297c10cac" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43681 } } peers { permanent_uuid: "770d5930d28448de9835853a4a24623c" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38345 } }
I20260620 23:16:43.780870 4411 ts_tablet_manager.cc:1428] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:16:43.780939 4264 heartbeater.cc:503] Master 127.16.79.62:36751 was elected leader, sending a full tablet report...
W20260620 23:16:43.851613 4265 tablet.cc:2369] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:16:43.877981 4415 raft_consensus.cc:482] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:16:43.878161 4415 raft_consensus.cc:504] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3286108e74b642298460f1b4dcbbe2b2" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36269 } } peers { permanent_uuid: "7acd38c33ba240c5a9ff2b0297c10cac" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43681 } } peers { permanent_uuid: "770d5930d28448de9835853a4a24623c" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38345 } }
I20260620 23:16:43.878815 4415 leader_election.cc:288] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7acd38c33ba240c5a9ff2b0297c10cac (127.16.79.2:43681), 770d5930d28448de9835853a4a24623c (127.16.79.3:38345)
I20260620 23:16:43.882411 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.879141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:48603 (local address 127.16.79.3:38345)
0620 23:16:43.879221 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.879224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.879239 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:16:43.879285 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.879286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.879295 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.879403 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:43.879664 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.880343 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.881665 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.882040 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.882125 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.882137 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.882290 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.882329 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:16:43.882419 4408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.879141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.79.1:46219 (local address 127.16.79.2:43681)
0620 23:16:43.879198 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.879200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.879487 (+ 287us) server_negotiation.cc:407] Connection header received
0620 23:16:43.879555 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.879556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.879571 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.879648 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:16:43.879834 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.880404 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.881690 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.882049 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.882135 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.882145 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.882313 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.882350 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:16:43.882423 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.879049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.3:38345 (local address 127.16.79.1:48603)
0620 23:16:43.879111 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:43.879124 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:43.879170 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:43.879493 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:43.879495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:43.879497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:43.879578 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.879581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.880525 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.880526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.881579 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.881582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.882135 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.882136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.882209 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.882212 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:43.882285 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:16:43.882333 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":66}
I20260620 23:16:43.882436 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.879049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.2:43681 (local address 127.16.79.1:46219)
0620 23:16:43.879405 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:43.879415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:43.879494 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:43.879659 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:43.879660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:43.879662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:16:43.879756 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.879759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.880525 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.880526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.881598 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:43.881601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.882138 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.882139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:43.882238 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.882241 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:16:43.882308 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:16:43.882347 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":178,"thread_start_us":121,"threads_started":1}
I20260620 23:16:43.882812 4218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969930bf84274399a76f6c62d6f4c43f" candidate_uuid: "3286108e74b642298460f1b4dcbbe2b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acd38c33ba240c5a9ff2b0297c10cac" is_pre_election: true
I20260620 23:16:43.882966 4218 raft_consensus.cc:2454] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3286108e74b642298460f1b4dcbbe2b2 in term 0.
I20260620 23:16:43.882969 4349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969930bf84274399a76f6c62d6f4c43f" candidate_uuid: "3286108e74b642298460f1b4dcbbe2b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770d5930d28448de9835853a4a24623c" is_pre_election: true
I20260620 23:16:43.883112 4349 raft_consensus.cc:2454] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3286108e74b642298460f1b4dcbbe2b2 in term 0.
I20260620 23:16:43.883204 4023 leader_election.cc:302] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286108e74b642298460f1b4dcbbe2b2, 7acd38c33ba240c5a9ff2b0297c10cac; no voters:
I20260620 23:16:43.883404 4415 raft_consensus.cc:2781] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:16:43.883476 4415 raft_consensus.cc:482] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:16:43.883512 4415 raft_consensus.cc:3037] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:43.884212 4415 raft_consensus.cc:504] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3286108e74b642298460f1b4dcbbe2b2" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36269 } } peers { permanent_uuid: "7acd38c33ba240c5a9ff2b0297c10cac" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43681 } } peers { permanent_uuid: "770d5930d28448de9835853a4a24623c" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38345 } }
I20260620 23:16:43.884371 4415 leader_election.cc:288] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [CANDIDATE]: Term 1 election: Requested vote from peers 7acd38c33ba240c5a9ff2b0297c10cac (127.16.79.2:43681), 770d5930d28448de9835853a4a24623c (127.16.79.3:38345)
I20260620 23:16:43.884531 4349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969930bf84274399a76f6c62d6f4c43f" candidate_uuid: "3286108e74b642298460f1b4dcbbe2b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770d5930d28448de9835853a4a24623c"
I20260620 23:16:43.884613 4349 raft_consensus.cc:3037] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:43.884560 4218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969930bf84274399a76f6c62d6f4c43f" candidate_uuid: "3286108e74b642298460f1b4dcbbe2b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acd38c33ba240c5a9ff2b0297c10cac"
I20260620 23:16:43.884665 4218 raft_consensus.cc:3037] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:43.885321 4349 raft_consensus.cc:2454] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286108e74b642298460f1b4dcbbe2b2 in term 1.
I20260620 23:16:43.885426 4218 raft_consensus.cc:2454] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286108e74b642298460f1b4dcbbe2b2 in term 1.
I20260620 23:16:43.885500 4022 leader_election.cc:302] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286108e74b642298460f1b4dcbbe2b2, 770d5930d28448de9835853a4a24623c; no voters:
I20260620 23:16:43.885588 4415 raft_consensus.cc:2781] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:43.885720 4415 raft_consensus.cc:686] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 1 LEADER]: Becoming Leader. State: Replica: 3286108e74b642298460f1b4dcbbe2b2, State: Running, Role: LEADER
I20260620 23:16:43.885812 4415 consensus_queue.cc:237] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3286108e74b642298460f1b4dcbbe2b2" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36269 } } peers { permanent_uuid: "7acd38c33ba240c5a9ff2b0297c10cac" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43681 } } peers { permanent_uuid: "770d5930d28448de9835853a4a24623c" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38345 } }
I20260620 23:16:43.886546 3945 catalog_manager.cc:5495] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3286108e74b642298460f1b4dcbbe2b2 (127.16.79.1). New cstate: current_term: 1 leader_uuid: "3286108e74b642298460f1b4dcbbe2b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3286108e74b642298460f1b4dcbbe2b2" member_type: VOTER last_known_addr { host: "127.16.79.1" port: 36269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7acd38c33ba240c5a9ff2b0297c10cac" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "770d5930d28448de9835853a4a24623c" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38345 } health_report { overall_health: UNKNOWN } } }
I20260620 23:16:43.915262 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:43.912510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.16.79.1:36269)
0620 23:16:43.912588 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:43.912590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:43.912598 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:16:43.912629 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:43.912630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:43.912650 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:43.912782 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:43.912924 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.913417 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.914141 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:43.914330 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:43.914393 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:43.914734 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:43.914857 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:43.914938 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:43.915108 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 23:16:43.915144 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:16:43.917763 4218 raft_consensus.cc:1261] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac [term 1 FOLLOWER]: Refusing update from remote peer 3286108e74b642298460f1b4dcbbe2b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:43.917763 4349 raft_consensus.cc:1261] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c [term 1 FOLLOWER]: Refusing update from remote peer 3286108e74b642298460f1b4dcbbe2b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:43.918260 4419 consensus_queue.cc:1035] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7acd38c33ba240c5a9ff2b0297c10cac" member_type: VOTER last_known_addr { host: "127.16.79.2" port: 43681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:43.918500 4415 consensus_queue.cc:1035] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "770d5930d28448de9835853a4a24623c" member_type: VOTER last_known_addr { host: "127.16.79.3" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:43.921197 4423 mvcc.cc:205] Tried to move back new op lower bound from 7299061366443167744 to 7299061366316818432. Current Snapshot: MvccSnapshot[applied={T|T < 7299061366443167744}]
I20260620 23:16:43.921483 4422 mvcc.cc:205] Tried to move back new op lower bound from 7299061366443167744 to 7299061366316818432. Current Snapshot: MvccSnapshot[applied={T|T < 7299061366443167744 or (T in {7299061366443167744})}]
I20260620 23:16:43.922127 4425 mvcc.cc:205] Tried to move back new op lower bound from 7299061366443167744 to 7299061366316818432. Current Snapshot: MvccSnapshot[applied={T|T < 7299061366443167744}]
I20260620 23:16:43.923453 3944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:16:43.923573 3944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:16:43.924603 3944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beaea6cbcfba48ddbb23c6f8b4495917: Sending DeleteTablet for 3 replicas of tablet 969930bf84274399a76f6c62d6f4c43f
I20260620 23:16:43.925098 4067 tablet_service.cc:1514] Processing DeleteTablet for tablet 969930bf84274399a76f6c62d6f4c43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:43 UTC) from {username='slave'} at 127.0.0.1:39388
I20260620 23:16:43.925134 4198 tablet_service.cc:1514] Processing DeleteTablet for tablet 969930bf84274399a76f6c62d6f4c43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:43 UTC) from {username='slave'} at 127.0.0.1:58740
I20260620 23:16:43.925359 4431 tablet_replica.cc:330] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2: stopping tablet replica
I20260620 23:16:43.925416 4329 tablet_service.cc:1514] Processing DeleteTablet for tablet 969930bf84274399a76f6c62d6f4c43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:16:43 UTC) from {username='slave'} at 127.0.0.1:60782
I20260620 23:16:43.925467 4431 raft_consensus.cc:2229] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:16:43.925472 4432 tablet_replica.cc:330] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac: stopping tablet replica
I20260620 23:16:43.925575 4431 raft_consensus.cc:2258] T 969930bf84274399a76f6c62d6f4c43f P 3286108e74b642298460f1b4dcbbe2b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:43.925782 4432 raft_consensus.cc:2229] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:16:43.925814 4433 tablet_replica.cc:330] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c: stopping tablet replica
I20260620 23:16:43.925915 4433 raft_consensus.cc:2229] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:16:43.926028 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 4005
I20260620 23:16:43.926028 4432 raft_consensus.cc:2258] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:43.926038 4433 raft_consensus.cc:2258] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:16:43.926601 4433 ts_tablet_manager.cc:1905] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:16:43.926640 4432 ts_tablet_manager.cc:1905] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:16:43.928373 4433 ts_tablet_manager.cc:1918] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:16:43.928531 4433 log.cc:1196] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-2/wal/wals/969930bf84274399a76f6c62d6f4c43f
I20260620 23:16:43.928668 4432 ts_tablet_manager.cc:1918] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:16:43.928779 4432 log.cc:1196] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac: Deleting WAL directory at /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997163808439-16700-0/minicluster-data/ts-1/wal/wals/969930bf84274399a76f6c62d6f4c43f
I20260620 23:16:43.928855 4433 ts_tablet_manager.cc:1939] T 969930bf84274399a76f6c62d6f4c43f P 770d5930d28448de9835853a4a24623c: Deleting consensus metadata
I20260620 23:16:43.929080 4432 ts_tablet_manager.cc:1939] T 969930bf84274399a76f6c62d6f4c43f P 7acd38c33ba240c5a9ff2b0297c10cac: Deleting consensus metadata
I20260620 23:16:43.929339 3931 catalog_manager.cc:4841] TS 770d5930d28448de9835853a4a24623c (127.16.79.3:38345): tablet 969930bf84274399a76f6c62d6f4c43f (table test-table [id=7a9d06098dc8417e899a90deb5ddda16]) successfully deleted
I20260620 23:16:43.929488 3932 catalog_manager.cc:4841] TS 7acd38c33ba240c5a9ff2b0297c10cac (127.16.79.2:43681): tablet 969930bf84274399a76f6c62d6f4c43f (table test-table [id=7a9d06098dc8417e899a90deb5ddda16]) successfully deleted
W20260620 23:16:43.933773 3929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.79.1:36269 (error 108)
W20260620 23:16:43.934284 4405 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:43.934148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.1:36269 (local address 127.0.0.1:39412)
0620 23:16:43.934207 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:43.934247 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.1:36269: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 23:16:43.934547 3929 catalog_manager.cc:4584] TS 3286108e74b642298460f1b4dcbbe2b2 (127.16.79.1:36269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 969930bf84274399a76f6c62d6f4c43f: Network error: Client connection negotiation failed: client connection to 127.16.79.1:36269: connect: Connection reset by peer (error 104)
I20260620 23:16:43.935027 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 4136
I20260620 23:16:43.940469 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 4267
I20260620 23:16:43.946414 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 3914
2026-06-20T23:16:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (810 ms)
[----------] 6 tests from EncryptionPolicyTest (8956 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:44 dist-test-slave-pswb krb5kdc[4437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:44 dist-test-slave-pswb krb5kdc[4437](info): set up 2 sockets
Jun 20 23:16:44 dist-test-slave-pswb krb5kdc[4437](info): commencing operation
krb5kdc: starting...
W20260620 23:16:45.009018 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:45 dist-test-slave-pswb krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:45.135555 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:45.217196 4450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:45.217424 4450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:45.217461 4450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:45.218588 4450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:45.218657 4450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:45.218683 4450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:45.218711 4450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:45.218740 4450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:45.221035 4450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39019
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:45.221318 4450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:45.221555 4450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:45.226616 4455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:45.226583 4456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:45.226662 4458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:45.227329 4450 server_base.cc:1034] running on GCE node
I20260620 23:16:45.227730 4450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:45.228114 4450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:45.229323 4450 hybrid_clock.cc:648] HybridClock initialized: now 1781997405229280 us; error 58 us; skew 500 ppm
Jun 20 23:16:45 dist-test-slave-pswb krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:45.231967 4450 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:45.232445 4450 webserver.cc:460] Webserver started at http://127.0.0.1:36821/ using document root <none> and password file <none>
I20260620 23:16:45.232681 4450 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:45.232810 4450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:45.232981 4450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:45.234063 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "d3d1148dc5734c86904e6223dd5c9bb3"
format_stamp: "Formatted at 2026-06-20 23:16:45 on dist-test-slave-pswb"
server_key: "38226996db7f1f72b1277192f358b769"
server_key_iv: "a29ccad021253661372def3016ca2c4f"
server_key_version: "encryptionkey@0"
I20260620 23:16:45.234442 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "d3d1148dc5734c86904e6223dd5c9bb3"
format_stamp: "Formatted at 2026-06-20 23:16:45 on dist-test-slave-pswb"
server_key: "38226996db7f1f72b1277192f358b769"
server_key_iv: "a29ccad021253661372def3016ca2c4f"
server_key_version: "encryptionkey@0"
I20260620 23:16:45.235826 4450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:16:45.236763 4465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:45.237036 4450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:16:45.237151 4450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "d3d1148dc5734c86904e6223dd5c9bb3"
format_stamp: "Formatted at 2026-06-20 23:16:45 on dist-test-slave-pswb"
server_key: "38226996db7f1f72b1277192f358b769"
server_key_iv: "a29ccad021253661372def3016ca2c4f"
server_key_version: "encryptionkey@0"
I20260620 23:16:45.237241 4450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:45.272497 4450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:45.274883 4450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:45.275071 4450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:45.279650 4450 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39019
I20260620 23:16:45.279681 4517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39019 every 8 connection(s)
I20260620 23:16:45.280053 4450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:45.281035 4518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:45.281055 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 4450
I20260620 23:16:45.281208 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:45.281451 16700 external_mini_cluster.cc:1428] Setting key 120843bcf15535589b0d5bb8d9729d43
I20260620 23:16:45.284242 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3: Bootstrap starting.
I20260620 23:16:45.284905 4518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:45.285197 4518 log.cc:824] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:45.285882 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3: No bootstrap required, opened a new log
I20260620 23:16:45.287875 4518 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d1148dc5734c86904e6223dd5c9bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } }
I20260620 23:16:45.288036 4518 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:45.288103 4518 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d1148dc5734c86904e6223dd5c9bb3, State: Initialized, Role: FOLLOWER
I20260620 23:16:45.288273 4518 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d1148dc5734c86904e6223dd5c9bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } }
I20260620 23:16:45.288372 4518 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:45.288420 4518 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:45.288473 4518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 23:16:45 dist-test-slave-pswb krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:16:45.289240 4518 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d1148dc5734c86904e6223dd5c9bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } }
I20260620 23:16:45.289343 4518 leader_election.cc:302] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3d1148dc5734c86904e6223dd5c9bb3; no voters:
I20260620 23:16:45.289532 4518 leader_election.cc:288] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:45.289593 4523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:45.289758 4523 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [term 1 LEADER]: Becoming Leader. State: Replica: d3d1148dc5734c86904e6223dd5c9bb3, State: Running, Role: LEADER
I20260620 23:16:45.289799 4518 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:45.289944 4523 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d1148dc5734c86904e6223dd5c9bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } }
I20260620 23:16:45.290387 4525 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3d1148dc5734c86904e6223dd5c9bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d1148dc5734c86904e6223dd5c9bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } } }
I20260620 23:16:45.290460 4525 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:45.291021 4531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:45.291479 4524 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3d1148dc5734c86904e6223dd5c9bb3. Latest consensus state: current_term: 1 leader_uuid: "d3d1148dc5734c86904e6223dd5c9bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d1148dc5734c86904e6223dd5c9bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } } }
I20260620 23:16:45.291545 4524 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:45.291877 4531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:45.293553 4531 catalog_manager.cc:1347] Generated new cluster ID: 7c78918635a84ff68bbfbf58c29b4a1b
I20260620 23:16:45.293602 4531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:45.294302 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:45.282177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44493 (local address 127.0.0.1:39019)
0620 23:16:45.282396 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:45.282408 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:45.282456 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:16:45.283039 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:45.283047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:45.283100 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:45.283246 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:45.284196 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:45.285429 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:45.286318 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:45.286760 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:45.286908 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:45.289968 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:45.289973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:45.289973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:45.289985 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:45.292378 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:45.292900 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:45.292902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:45.292902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:45.293011 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:45.293281 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:45.293282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:45.293282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:45.293592 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:45.293740 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:45.294018 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 23:16:45.294112 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":113,"threads_started":1}
I20260620 23:16:45.295383 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:45.308306 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:45.296814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60709 (local address 127.0.0.1:39019)
0620 23:16:45.296845 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:45.296847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:45.301159 (+ 4312us) server_negotiation.cc:407] Connection header received
0620 23:16:45.301221 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:45.301222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:45.301231 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:45.301284 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:45.301988 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:45.302690 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:45.303710 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:45.304030 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:45.304086 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:45.305565 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:45.305566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:45.305567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:45.305573 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:45.307074 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:45.307502 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:45.307503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:45.307503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:45.307553 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:45.307797 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:45.307797 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:45.307798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:45.307913 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:45.308007 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:45.308161 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:16:45.308219 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":16}
I20260620 23:16:45.318564 4531 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:45.319226 4531 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:45.327665 4531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3d1148dc5734c86904e6223dd5c9bb3: Generated new TSK 0
I20260620 23:16:45.327867 4531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:45.335770 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 4450
W20260620 23:16:45.339663 4435 connection.cc:535] client connection to 127.0.0.1:39019 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39019): Connection reset by peer (error 104)
2026-06-20T23:16:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:45 dist-test-slave-pswb krb5kdc[4549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:45 dist-test-slave-pswb krb5kdc[4549](info): set up 2 sockets
Jun 20 23:16:45 dist-test-slave-pswb krb5kdc[4549](info): commencing operation
krb5kdc: starting...
W20260620 23:16:46.380674 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:46 dist-test-slave-pswb krb5kdc[4549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:46.495713 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:46.576092 4562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:46.576285 4562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:46.576308 4562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:46.577433 4562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:46.577479 4562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:46.577494 4562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:46.577507 4562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:46.577519 4562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:46.578886 4562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39111
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:46.579066 4562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:46.579222 4562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:46.584084 4568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:46.584084 4570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:46.584161 4567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:46.584311 4562 server_base.cc:1034] running on GCE node
I20260620 23:16:46.584545 4562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:46.584856 4562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:46.586018 4562 hybrid_clock.cc:648] HybridClock initialized: now 1781997406585997 us; error 37 us; skew 500 ppm
Jun 20 23:16:46 dist-test-slave-pswb krb5kdc[4549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:46.588406 4562 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:46.588948 4562 webserver.cc:460] Webserver started at http://127.0.0.1:44843/ using document root <none> and password file <none>
I20260620 23:16:46.589195 4562 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:46.589265 4562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:46.589421 4562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:46.590427 4562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "369850992ca943e5809ba0baf61fbd5b"
format_stamp: "Formatted at 2026-06-20 23:16:46 on dist-test-slave-pswb"
server_key: "ed574907f50d1168cdf91d9fa160cb3e"
server_key_iv: "ba67924326af9ce08d5e33213a28347e"
server_key_version: "encryptionkey@0"
I20260620 23:16:46.590775 4562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "369850992ca943e5809ba0baf61fbd5b"
format_stamp: "Formatted at 2026-06-20 23:16:46 on dist-test-slave-pswb"
server_key: "ed574907f50d1168cdf91d9fa160cb3e"
server_key_iv: "ba67924326af9ce08d5e33213a28347e"
server_key_version: "encryptionkey@0"
I20260620 23:16:46.592123 4562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:16:46.592892 4577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:46.593279 4562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:46.593389 4562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "369850992ca943e5809ba0baf61fbd5b"
format_stamp: "Formatted at 2026-06-20 23:16:46 on dist-test-slave-pswb"
server_key: "ed574907f50d1168cdf91d9fa160cb3e"
server_key_iv: "ba67924326af9ce08d5e33213a28347e"
server_key_version: "encryptionkey@0"
I20260620 23:16:46.593482 4562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:46.612329 4562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:46.615001 4562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:46.615201 4562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:46.619604 4562 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39111
I20260620 23:16:46.619648 4629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39111 every 8 connection(s)
I20260620 23:16:46.619993 4562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:46.620679 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 4562
I20260620 23:16:46.620846 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:46.620932 4630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:46.621078 16700 external_mini_cluster.cc:1428] Setting key c77d632ddf273b42e7d337b58b4ae114
I20260620 23:16:46.624256 4630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b: Bootstrap starting.
I20260620 23:16:46.624909 4630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:46.625217 4630 log.cc:824] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:46.625931 4630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b: No bootstrap required, opened a new log
I20260620 23:16:46.627419 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:46.621585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33903 (local address 127.0.0.1:39111)
0620 23:16:46.621781 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:46.621790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:46.621822 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:16:46.622500 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:46.622509 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:46.622569 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:46.622731 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:46.622919 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:46.624247 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:46.625110 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:46.625666 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:46.625779 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:46.626767 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:46.626771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:46.626772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:46.626975 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:46.627012 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:46.627026 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:46.627109 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:46.627193 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 23:16:46.627219 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260620 23:16:46.628417 4630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369850992ca943e5809ba0baf61fbd5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39111 } }
I20260620 23:16:46.628569 4630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:46.628603 4630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 369850992ca943e5809ba0baf61fbd5b, State: Initialized, Role: FOLLOWER
I20260620 23:16:46.628732 4630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369850992ca943e5809ba0baf61fbd5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39111 } }
I20260620 23:16:46.628829 4630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:46.628825 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:46.628865 4630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:46.628916 4630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:46.629585 4630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369850992ca943e5809ba0baf61fbd5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39111 } }
I20260620 23:16:46.629696 4630 leader_election.cc:302] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 369850992ca943e5809ba0baf61fbd5b; no voters:
I20260620 23:16:46.629845 4630 leader_election.cc:288] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:46.629990 4635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:46.630148 4635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [term 1 LEADER]: Becoming Leader. State: Replica: 369850992ca943e5809ba0baf61fbd5b, State: Running, Role: LEADER
I20260620 23:16:46.630168 4630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:46.630262 4635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369850992ca943e5809ba0baf61fbd5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39111 } }
I20260620 23:16:46.630568 4635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 369850992ca943e5809ba0baf61fbd5b. Latest consensus state: current_term: 1 leader_uuid: "369850992ca943e5809ba0baf61fbd5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369850992ca943e5809ba0baf61fbd5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39111 } } }
I20260620 23:16:46.630642 4635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:46.630601 4641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "369850992ca943e5809ba0baf61fbd5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369850992ca943e5809ba0baf61fbd5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39111 } } }
I20260620 23:16:46.630837 4641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b [sys.catalog]: This master's current role is: LEADER
W20260620 23:16:46.631855 4655 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:16:46.631958 4655 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:16:46.632047 4646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:46.632257 4646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:46.634521 4646 catalog_manager.cc:1347] Generated new cluster ID: 5ddff819f8ce42f6a8883a4c439b4a03
I20260620 23:16:46.634568 4646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:46.636204 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:46.631885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41003 (local address 127.0.0.1:39111)
0620 23:16:46.631923 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:46.631925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:46.632210 (+ 285us) server_negotiation.cc:407] Connection header received
0620 23:16:46.632232 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:46.632232 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:46.632242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:46.632293 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:46.632474 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:46.633195 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:46.634204 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:46.634804 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:46.635037 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:46.635890 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:46.635892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:46.635892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:46.635898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:46.635916 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:46.635920 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:46.635998 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:46.636074 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 23:16:46.636086 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":18}
I20260620 23:16:46.647710 4646 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:46.648288 4646 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:46.655609 4646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 369850992ca943e5809ba0baf61fbd5b: Generated new TSK 0
I20260620 23:16:46.655812 4646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:46.664048 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 4562
W20260620 23:16:46.668078 4547 connection.cc:535] client connection to 127.0.0.1:39111 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39111): Connection reset by peer (error 104)
2026-06-20T23:16:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:46 dist-test-slave-pswb krb5kdc[4661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:46 dist-test-slave-pswb krb5kdc[4661](info): set up 2 sockets
Jun 20 23:16:46 dist-test-slave-pswb krb5kdc[4661](info): commencing operation
krb5kdc: starting...
W20260620 23:16:47.737574 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:47 dist-test-slave-pswb krb5kdc[4661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:47.860458 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:47.938614 4674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:47.938845 4674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:47.938884 4674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:47.940011 4674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:47.940078 4674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:47.940101 4674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:47.940130 4674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:47.940153 4674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:47.941650 4674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:47.941917 4674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:47.942130 4674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:47.947189 4679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:47.947202 4680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:47.947372 4674 server_base.cc:1034] running on GCE node
W20260620 23:16:47.947579 4682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:47.947861 4674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:47.948146 4674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:47.949347 4674 hybrid_clock.cc:648] HybridClock initialized: now 1781997407949295 us; error 105 us; skew 500 ppm
Jun 20 23:16:47 dist-test-slave-pswb krb5kdc[4661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:47.951875 4674 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:47.952283 4674 webserver.cc:460] Webserver started at http://127.0.0.1:46387/ using document root <none> and password file <none>
I20260620 23:16:47.952533 4674 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:47.952601 4674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:47.952770 4674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:47.953816 4674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "ed2cd625d0cc46a39e9aabe3774133e9"
format_stamp: "Formatted at 2026-06-20 23:16:47 on dist-test-slave-pswb"
server_key: "1e8367b2ce132f3b1c38f5153864c183"
server_key_iv: "92cbcfa9345478a5d9755849822bb074"
server_key_version: "encryptionkey@0"
I20260620 23:16:47.954171 4674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "ed2cd625d0cc46a39e9aabe3774133e9"
format_stamp: "Formatted at 2026-06-20 23:16:47 on dist-test-slave-pswb"
server_key: "1e8367b2ce132f3b1c38f5153864c183"
server_key_iv: "92cbcfa9345478a5d9755849822bb074"
server_key_version: "encryptionkey@0"
I20260620 23:16:47.955487 4674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:16:47.956153 4689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:47.956317 4674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:16:47.956415 4674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "ed2cd625d0cc46a39e9aabe3774133e9"
format_stamp: "Formatted at 2026-06-20 23:16:47 on dist-test-slave-pswb"
server_key: "1e8367b2ce132f3b1c38f5153864c183"
server_key_iv: "92cbcfa9345478a5d9755849822bb074"
server_key_version: "encryptionkey@0"
I20260620 23:16:47.956496 4674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:47.959849 4674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:47.962212 4674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:47.962366 4674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:47.966686 4674 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44329
I20260620 23:16:47.966900 4741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44329 every 8 connection(s)
I20260620 23:16:47.967089 4674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:47.968255 4742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:47.971159 4742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9: Bootstrap starting.
I20260620 23:16:47.971982 4742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:47.972517 4742 log.cc:824] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:47.973403 4742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9: No bootstrap required, opened a new log
I20260620 23:16:47.974906 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 4674
I20260620 23:16:47.975024 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:47.975210 16700 external_mini_cluster.cc:1428] Setting key 34a94d98e43905113612df3f124eeba9
I20260620 23:16:47.975610 4742 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2cd625d0cc46a39e9aabe3774133e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44329 } }
I20260620 23:16:47.975816 4742 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:47.975905 4742 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2cd625d0cc46a39e9aabe3774133e9, State: Initialized, Role: FOLLOWER
I20260620 23:16:47.976058 4742 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2cd625d0cc46a39e9aabe3774133e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44329 } }
I20260620 23:16:47.976135 4742 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:47.976157 4742 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:47.976222 4742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:47.977378 4746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:47.975729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55737 (local address 127.0.0.1:44329)
0620 23:16:47.975948 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:47.975957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:47.975990 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:16:47.976101 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:47.976108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:47.976161 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:47.976290 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:47.976936 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:47.976939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:47.976939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:47.976943 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:47.976978 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:47.976988 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:47.977041 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:47.977126 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 23:16:47.977180 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":108,"threads_started":1}
I20260620 23:16:47.977533 4742 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2cd625d0cc46a39e9aabe3774133e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44329 } }
I20260620 23:16:47.977635 4742 leader_election.cc:302] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed2cd625d0cc46a39e9aabe3774133e9; no voters:
I20260620 23:16:47.977784 4742 leader_election.cc:288] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:47.977808 4747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:47.977996 4747 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [term 1 LEADER]: Becoming Leader. State: Replica: ed2cd625d0cc46a39e9aabe3774133e9, State: Running, Role: LEADER
I20260620 23:16:47.978070 4742 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:47.978127 4747 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2cd625d0cc46a39e9aabe3774133e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44329 } }
I20260620 23:16:47.978283 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:47.978493 4747 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2cd625d0cc46a39e9aabe3774133e9. Latest consensus state: current_term: 1 leader_uuid: "ed2cd625d0cc46a39e9aabe3774133e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2cd625d0cc46a39e9aabe3774133e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44329 } } }
I20260620 23:16:47.978502 4749 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed2cd625d0cc46a39e9aabe3774133e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2cd625d0cc46a39e9aabe3774133e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44329 } } }
I20260620 23:16:47.978579 4747 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:47.978583 4749 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:47.979254 4755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:47.979880 4755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:47.981374 4755 catalog_manager.cc:1347] Generated new cluster ID: d9614b01dc7b4e2e84311be25ac2d7f9
I20260620 23:16:47.981426 4755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:47.985807 4746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:47.979798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42295 (local address 127.0.0.1:44329)
0620 23:16:47.979836 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:47.979838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:47.984886 (+ 5048us) server_negotiation.cc:407] Connection header received
0620 23:16:47.984962 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:47.984963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:47.984969 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:47.985029 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:47.985613 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:47.985614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:47.985614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:47.985617 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:47.985628 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:47.985632 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:47.985678 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:47.985731 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 23:16:47.985741 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:16:47.990744 4755 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:47.990890 4755 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:48.005035 4755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed2cd625d0cc46a39e9aabe3774133e9: Generated new TSK 0
I20260620 23:16:48.005336 4755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:48.012162 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 4674
2026-06-20T23:16:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:48 dist-test-slave-pswb krb5kdc[4773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:48 dist-test-slave-pswb krb5kdc[4773](info): set up 2 sockets
Jun 20 23:16:48 dist-test-slave-pswb krb5kdc[4773](info): commencing operation
krb5kdc: starting...
W20260620 23:16:49.056941 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:49 dist-test-slave-pswb krb5kdc[4773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:49.184373 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:49.265017 4786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:49.265223 4786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:49.265244 4786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:49.266278 4786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:49.266317 4786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:49.266331 4786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:49.266345 4786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:49.266356 4786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:49.267704 4786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:49.267879 4786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:49.268042 4786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:49.273027 4792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:49.273005 4791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:49.273185 4786 server_base.cc:1034] running on GCE node
W20260620 23:16:49.273010 4794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:49.273546 4786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:49.273831 4786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:49.275015 4786 hybrid_clock.cc:648] HybridClock initialized: now 1781997409275013 us; error 50 us; skew 500 ppm
Jun 20 23:16:49 dist-test-slave-pswb krb5kdc[4773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:49.277753 4786 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:49.278184 4786 webserver.cc:460] Webserver started at http://127.0.0.1:39193/ using document root <none> and password file <none>
I20260620 23:16:49.278416 4786 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:49.278471 4786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:49.278556 4786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:49.279502 4786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "da180135aee04fa99653b31739c1df91"
format_stamp: "Formatted at 2026-06-20 23:16:49 on dist-test-slave-pswb"
server_key: "0a49b6dadc606c70f1a0752ec00b7739"
server_key_iv: "9c29118a1d7d0fbbff3a7fa13f774ebd"
server_key_version: "encryptionkey@0"
I20260620 23:16:49.279801 4786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "da180135aee04fa99653b31739c1df91"
format_stamp: "Formatted at 2026-06-20 23:16:49 on dist-test-slave-pswb"
server_key: "0a49b6dadc606c70f1a0752ec00b7739"
server_key_iv: "9c29118a1d7d0fbbff3a7fa13f774ebd"
server_key_version: "encryptionkey@0"
I20260620 23:16:49.281016 4786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:16:49.281699 4801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:49.281883 4786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:16:49.281972 4786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "da180135aee04fa99653b31739c1df91"
format_stamp: "Formatted at 2026-06-20 23:16:49 on dist-test-slave-pswb"
server_key: "0a49b6dadc606c70f1a0752ec00b7739"
server_key_iv: "9c29118a1d7d0fbbff3a7fa13f774ebd"
server_key_version: "encryptionkey@0"
I20260620 23:16:49.282042 4786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:49.297904 4786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:49.300128 4786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:49.300294 4786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:49.305159 4786 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40257
I20260620 23:16:49.305200 4853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40257 every 8 connection(s)
I20260620 23:16:49.305648 4786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:49.306576 4854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:49.308897 4854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91: Bootstrap starting.
I20260620 23:16:49.309041 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 4786
I20260620 23:16:49.309171 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:49.309396 16700 external_mini_cluster.cc:1428] Setting key 20639cf0f64a465adb8a5f04ea215d13
I20260620 23:16:49.309633 4854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:49.309958 4854 log.cc:824] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:49.310889 4854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91: No bootstrap required, opened a new log
I20260620 23:16:49.313844 4854 raft_consensus.cc:348] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da180135aee04fa99653b31739c1df91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40257 } }
I20260620 23:16:49.314079 4854 raft_consensus.cc:374] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:49.314118 4854 raft_consensus.cc:729] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da180135aee04fa99653b31739c1df91, State: Initialized, Role: FOLLOWER
I20260620 23:16:49.314265 4854 consensus_queue.cc:260] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da180135aee04fa99653b31739c1df91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40257 } }
I20260620 23:16:49.314345 4854 raft_consensus.cc:388] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:49.314374 4854 raft_consensus.cc:482] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:49.314414 4854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:49.315557 4854 raft_consensus.cc:504] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da180135aee04fa99653b31739c1df91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40257 } }
I20260620 23:16:49.315742 4854 leader_election.cc:302] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da180135aee04fa99653b31739c1df91; no voters:
I20260620 23:16:49.315936 4854 leader_election.cc:288] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:49.315961 4859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:49.316072 4859 raft_consensus.cc:686] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [term 1 LEADER]: Becoming Leader. State: Replica: da180135aee04fa99653b31739c1df91, State: Running, Role: LEADER
I20260620 23:16:49.316215 4854 sys_catalog.cc:564] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:49.316226 4859 consensus_queue.cc:237] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da180135aee04fa99653b31739c1df91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40257 } }
I20260620 23:16:49.316609 4859 sys_catalog.cc:455] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da180135aee04fa99653b31739c1df91. Latest consensus state: current_term: 1 leader_uuid: "da180135aee04fa99653b31739c1df91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da180135aee04fa99653b31739c1df91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40257 } } }
I20260620 23:16:49.316877 4859 sys_catalog.cc:458] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:49.316795 4860 sys_catalog.cc:455] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da180135aee04fa99653b31739c1df91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da180135aee04fa99653b31739c1df91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40257 } } }
I20260620 23:16:49.317297 4860 sys_catalog.cc:458] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:49.317497 4868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 23:16:49 dist-test-slave-pswb krb5kdc[4773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:16:49.317987 4868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:49.319548 4868 catalog_manager.cc:1347] Generated new cluster ID: cbf25b4c7b2746908a980920cec1d7fa
I20260620 23:16:49.319597 4868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:49.321753 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:49.310039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55135 (local address 127.0.0.1:40257)
0620 23:16:49.310252 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:49.310262 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:49.310380 (+ 118us) server_negotiation.cc:407] Connection header received
0620 23:16:49.311257 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:49.311266 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:49.311320 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:49.311436 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:49.312514 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:49.313498 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:49.314657 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:49.315065 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:49.315158 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:49.318293 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:49.318297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:49.318298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:49.318309 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:49.320223 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:49.320599 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:49.320602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:49.320602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:49.320730 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:49.320926 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:49.320927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:49.320927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:49.321156 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:49.321332 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:49.321406 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 23:16:49.321541 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260620 23:16:49.325686 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:49.339910 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:49.329070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60241 (local address 127.0.0.1:40257)
0620 23:16:49.329111 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:49.329112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:49.332884 (+ 3772us) server_negotiation.cc:407] Connection header received
0620 23:16:49.332917 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:49.332918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:49.332927 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:49.332988 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:49.333617 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:49.334324 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:49.335206 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:49.335462 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:49.335667 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:49.337303 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:49.337305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:49.337305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:49.337314 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:49.338773 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:49.339133 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:49.339134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:49.339134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:49.339192 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:49.339395 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:49.339396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:49.339397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:49.339508 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:49.339606 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:49.339716 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 23:16:49.339802 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:16:49.343787 4868 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:49.344460 4868 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:49.351732 4868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da180135aee04fa99653b31739c1df91: Generated new TSK 0
I20260620 23:16:49.351897 4868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:49.359532 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 4786
2026-06-20T23:16:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:49 dist-test-slave-pswb krb5kdc[4885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:49 dist-test-slave-pswb krb5kdc[4885](info): set up 2 sockets
Jun 20 23:16:49 dist-test-slave-pswb krb5kdc[4885](info): commencing operation
krb5kdc: starting...
W20260620 23:16:50.421001 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:50 dist-test-slave-pswb krb5kdc[4885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:50.543637 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:50.629356 4898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:50.629594 4898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:50.629630 4898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:50.630777 4898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:50.630851 4898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:50.630875 4898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:50.630904 4898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:50.630935 4898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:50.632349 4898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:50.632632 4898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:50.632874 4898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:50.637717 4906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:50.637730 4904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:50.637769 4903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:50.637995 4898 server_base.cc:1034] running on GCE node
I20260620 23:16:50.638252 4898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:50.638530 4898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:50.639717 4898 hybrid_clock.cc:648] HybridClock initialized: now 1781997410639717 us; error 43 us; skew 500 ppm
Jun 20 23:16:50 dist-test-slave-pswb krb5kdc[4885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:50.642848 4898 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:50.643553 4898 webserver.cc:460] Webserver started at http://127.0.0.1:44863/ using document root <none> and password file <none>
I20260620 23:16:50.643792 4898 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:50.643842 4898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:50.643949 4898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:50.645233 4898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "1d9cd5cb8d874be08e7c2154ddc308f5"
format_stamp: "Formatted at 2026-06-20 23:16:50 on dist-test-slave-pswb"
server_key: "08cfa699c0fcbad34fc814fbe67dabb3"
server_key_iv: "7d59c23b3aef061c25362d0b106b41b3"
server_key_version: "encryptionkey@0"
I20260620 23:16:50.645573 4898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "1d9cd5cb8d874be08e7c2154ddc308f5"
format_stamp: "Formatted at 2026-06-20 23:16:50 on dist-test-slave-pswb"
server_key: "08cfa699c0fcbad34fc814fbe67dabb3"
server_key_iv: "7d59c23b3aef061c25362d0b106b41b3"
server_key_version: "encryptionkey@0"
I20260620 23:16:50.647039 4898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:16:50.647922 4913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:50.648242 4898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:50.648330 4898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "1d9cd5cb8d874be08e7c2154ddc308f5"
format_stamp: "Formatted at 2026-06-20 23:16:50 on dist-test-slave-pswb"
server_key: "08cfa699c0fcbad34fc814fbe67dabb3"
server_key_iv: "7d59c23b3aef061c25362d0b106b41b3"
server_key_version: "encryptionkey@0"
I20260620 23:16:50.648450 4898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:50.656248 4898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:50.658159 4898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:50.658259 4898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:50.661983 4898 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38725
I20260620 23:16:50.662032 4965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38725 every 8 connection(s)
I20260620 23:16:50.662341 4898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:50.663192 4966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:50.665486 4966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5: Bootstrap starting.
I20260620 23:16:50.665998 4966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:50.666224 4966 log.cc:824] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:50.667119 4966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5: No bootstrap required, opened a new log
I20260620 23:16:50.668457 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 4898
I20260620 23:16:50.668609 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:50.668879 16700 external_mini_cluster.cc:1428] Setting key 22e58cb3ead690f965e23ed1cc578199
I20260620 23:16:50.669750 4966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9cd5cb8d874be08e7c2154ddc308f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260620 23:16:50.669970 4966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:50.670040 4966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9cd5cb8d874be08e7c2154ddc308f5, State: Initialized, Role: FOLLOWER
I20260620 23:16:50.670213 4966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9cd5cb8d874be08e7c2154ddc308f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260620 23:16:50.670323 4966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:50.670356 4966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:50.670400 4966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:50.671340 4966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9cd5cb8d874be08e7c2154ddc308f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260620 23:16:50.671458 4966 leader_election.cc:302] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d9cd5cb8d874be08e7c2154ddc308f5; no voters:
I20260620 23:16:50.671645 4966 leader_election.cc:288] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:50.671720 4971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:50.671929 4966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:50.671936 4971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [term 1 LEADER]: Becoming Leader. State: Replica: 1d9cd5cb8d874be08e7c2154ddc308f5, State: Running, Role: LEADER
I20260620 23:16:50.672086 4971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9cd5cb8d874be08e7c2154ddc308f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260620 23:16:50.672574 4972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d9cd5cb8d874be08e7c2154ddc308f5. Latest consensus state: current_term: 1 leader_uuid: "1d9cd5cb8d874be08e7c2154ddc308f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9cd5cb8d874be08e7c2154ddc308f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } } }
I20260620 23:16:50.672673 4972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:50.672911 4980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:50.673499 4980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:50.675582 4980 catalog_manager.cc:1347] Generated new cluster ID: 37e2ec86ee754fa2b27058616ec622d2
I20260620 23:16:50.675639 4980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:50.676194 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:50.669505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53267 (local address 127.0.0.1:38725)
0620 23:16:50.669696 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:50.669703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:50.669734 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:16:50.669846 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:50.669852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:50.669894 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:50.670040 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:50.670313 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:50.672157 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:50.673440 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:50.674068 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:50.674186 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:50.675200 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:50.675204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:50.675204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:50.675432 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:50.675475 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:50.675492 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:50.675576 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:50.675901 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:16:50.675979 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20260620 23:16:50.675637 4973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d9cd5cb8d874be08e7c2154ddc308f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9cd5cb8d874be08e7c2154ddc308f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } } }
I20260620 23:16:50.677157 4973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:50.677201 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:50.685959 4980 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:50.686583 4980 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:50.689276 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:50.678701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53009 (local address 127.0.0.1:38725)
0620 23:16:50.678734 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:50.678736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:50.685663 (+ 6927us) server_negotiation.cc:407] Connection header received
0620 23:16:50.685720 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:50.685721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:50.685733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:50.685780 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:50.685987 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:50.686679 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:50.687739 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:50.688018 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:50.688078 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:50.688803 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:50.688805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:50.688805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:50.688809 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:50.688825 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:50.688830 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:50.688880 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:50.689110 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 23:16:50.689184 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:16:50.698271 4980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d9cd5cb8d874be08e7c2154ddc308f5: Generated new TSK 0
I20260620 23:16:50.698463 4980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:50.702597 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 4898
2026-06-20T23:16:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:50 dist-test-slave-pswb krb5kdc[4997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:50 dist-test-slave-pswb krb5kdc[4997](info): set up 2 sockets
Jun 20 23:16:50 dist-test-slave-pswb krb5kdc[4997](info): commencing operation
krb5kdc: starting...
W20260620 23:16:51.744897 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:51 dist-test-slave-pswb krb5kdc[4997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:51.862186 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:51.944164 5010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:51.944355 5010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:51.944377 5010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:51.945520 5010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:51.945569 5010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:51.945585 5010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:51.945597 5010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:51.945611 5010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:51.946978 5010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:51.947163 5010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:51.947325 5010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:51.951865 5018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:51.952028 5010 server_base.cc:1034] running on GCE node
W20260620 23:16:51.951891 5016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:51.951939 5015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:51.952344 5010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:51.952625 5010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:51.953805 5010 hybrid_clock.cc:648] HybridClock initialized: now 1781997411953783 us; error 39 us; skew 500 ppm
Jun 20 23:16:51 dist-test-slave-pswb krb5kdc[4997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:51.956214 5010 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:51.956616 5010 webserver.cc:460] Webserver started at http://127.0.0.1:41381/ using document root <none> and password file <none>
I20260620 23:16:51.956879 5010 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:51.956925 5010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:51.957010 5010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:51.957953 5010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "f447e5b0e37e46bb9b8d32d61e74810a"
format_stamp: "Formatted at 2026-06-20 23:16:51 on dist-test-slave-pswb"
server_key: "2196b379c0c5c07e2e48c0b74bfd354e"
server_key_iv: "bb8807eb59d004b3891becb6ca1ad952"
server_key_version: "encryptionkey@0"
I20260620 23:16:51.958230 5010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "f447e5b0e37e46bb9b8d32d61e74810a"
format_stamp: "Formatted at 2026-06-20 23:16:51 on dist-test-slave-pswb"
server_key: "2196b379c0c5c07e2e48c0b74bfd354e"
server_key_iv: "bb8807eb59d004b3891becb6ca1ad952"
server_key_version: "encryptionkey@0"
I20260620 23:16:51.959673 5010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:16:51.960362 5025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:51.960654 5010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:51.960768 5010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "f447e5b0e37e46bb9b8d32d61e74810a"
format_stamp: "Formatted at 2026-06-20 23:16:51 on dist-test-slave-pswb"
server_key: "2196b379c0c5c07e2e48c0b74bfd354e"
server_key_iv: "bb8807eb59d004b3891becb6ca1ad952"
server_key_version: "encryptionkey@0"
I20260620 23:16:51.960845 5010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:51.964064 5010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:51.966637 5010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:51.966745 5010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:51.971565 5010 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40797
I20260620 23:16:51.971593 5077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40797 every 8 connection(s)
I20260620 23:16:51.972013 5010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:51.972972 5078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:51.975078 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a: Bootstrap starting.
I20260620 23:16:51.975615 5078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:51.975890 5078 log.cc:824] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:51.976538 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a: No bootstrap required, opened a new log
I20260620 23:16:51.976727 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 5010
I20260620 23:16:51.976861 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:51.977051 16700 external_mini_cluster.cc:1428] Setting key 0bbc9953eaefea540462ea9d61d71f64
I20260620 23:16:51.978861 5078 raft_consensus.cc:348] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f447e5b0e37e46bb9b8d32d61e74810a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40797 } }
I20260620 23:16:51.979058 5078 raft_consensus.cc:374] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:51.979091 5078 raft_consensus.cc:729] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f447e5b0e37e46bb9b8d32d61e74810a, State: Initialized, Role: FOLLOWER
I20260620 23:16:51.979205 5078 consensus_queue.cc:260] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f447e5b0e37e46bb9b8d32d61e74810a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40797 } }
I20260620 23:16:51.979285 5078 raft_consensus.cc:388] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:51.979386 5078 raft_consensus.cc:482] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:51.979449 5078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:51.980268 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:51.977493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51367 (local address 127.0.0.1:40797)
0620 23:16:51.977690 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:51.977699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:51.977733 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:16:51.979059 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:51.979067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:51.979102 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:51.979246 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:51.979873 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:51.979877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:51.979877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:51.979881 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:51.979925 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:51.979935 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:51.980006 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:51.980076 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 23:16:51.980116 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260620 23:16:51.980341 5078 raft_consensus.cc:504] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f447e5b0e37e46bb9b8d32d61e74810a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40797 } }
I20260620 23:16:51.980454 5078 leader_election.cc:302] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f447e5b0e37e46bb9b8d32d61e74810a; no voters:
I20260620 23:16:51.980594 5078 leader_election.cc:288] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:51.980640 5083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:51.980866 5078 sys_catalog.cc:564] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:51.980890 5083 raft_consensus.cc:686] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [term 1 LEADER]: Becoming Leader. State: Replica: f447e5b0e37e46bb9b8d32d61e74810a, State: Running, Role: LEADER
I20260620 23:16:51.981060 5083 consensus_queue.cc:237] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f447e5b0e37e46bb9b8d32d61e74810a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40797 } }
I20260620 23:16:51.981163 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:51.982151 5085 sys_catalog.cc:455] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f447e5b0e37e46bb9b8d32d61e74810a. Latest consensus state: current_term: 1 leader_uuid: "f447e5b0e37e46bb9b8d32d61e74810a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f447e5b0e37e46bb9b8d32d61e74810a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40797 } } }
I20260620 23:16:51.982235 5085 sys_catalog.cc:458] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [sys.catalog]: This master's current role is: LEADER
W20260620 23:16:51.982762 5098 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:16:51.982877 5098 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:16:51.982918 5084 sys_catalog.cc:455] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f447e5b0e37e46bb9b8d32d61e74810a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f447e5b0e37e46bb9b8d32d61e74810a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40797 } } }
I20260620 23:16:51.982981 5102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:51.983247 5102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:51.983076 5084 sys_catalog.cc:458] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:51.984016 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:51.982682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41235 (local address 127.0.0.1:40797)
0620 23:16:51.982717 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:51.982719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:51.982939 (+ 220us) server_negotiation.cc:407] Connection header received
0620 23:16:51.983040 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:51.983041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:51.983045 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:51.983119 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:51.983779 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:51.983781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:51.983781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:51.983785 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:51.983798 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:51.983802 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:51.983832 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:51.983944 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 23:16:51.983956 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:16:51.984850 5102 catalog_manager.cc:1347] Generated new cluster ID: 6e2515304abe427da2e074ecb84e7cb0
I20260620 23:16:51.984899 5102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:51.997210 5102 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:51.997478 5102 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:52.006311 5102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f447e5b0e37e46bb9b8d32d61e74810a: Generated new TSK 0
I20260620 23:16:52.006484 5102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:52.009059 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 5010
2026-06-20T23:16:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:52 dist-test-slave-pswb krb5kdc[5109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:52 dist-test-slave-pswb krb5kdc[5109](info): set up 2 sockets
Jun 20 23:16:52 dist-test-slave-pswb krb5kdc[5109](info): commencing operation
krb5kdc: starting...
W20260620 23:16:53.057051 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:53 dist-test-slave-pswb krb5kdc[5109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:53.175271 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33657
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:53.255270 5122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:53.255517 5122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:53.255554 5122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:53.257119 5122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:53.257167 5122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:53.257187 5122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:53.257210 5122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:53.257239 5122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:53.258673 5122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:53.258930 5122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:53.259136 5122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:53.264159 5130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:53.264019 5128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:53.264052 5122 server_base.cc:1034] running on GCE node
W20260620 23:16:53.264019 5127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:53.264874 5122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:53.265292 5122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:53.266489 5122 hybrid_clock.cc:648] HybridClock initialized: now 1781997413266465 us; error 40 us; skew 500 ppm
Jun 20 23:16:53 dist-test-slave-pswb krb5kdc[5109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:53.269115 5122 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:53.269814 5122 webserver.cc:460] Webserver started at http://127.0.0.1:32789/ using document root <none> and password file <none>
I20260620 23:16:53.270135 5122 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:53.270215 5122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:53.270385 5122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:53.271731 5122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "f65162096925476fbc3616c9df814a88"
format_stamp: "Formatted at 2026-06-20 23:16:53 on dist-test-slave-pswb"
server_key: "fa29230d068e24de99eb04190a5f77cc"
server_key_iv: "ac1b4c5a2fe96f61fe7a63e5519d6432"
server_key_version: "encryptionkey@0"
I20260620 23:16:53.272161 5122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "f65162096925476fbc3616c9df814a88"
format_stamp: "Formatted at 2026-06-20 23:16:53 on dist-test-slave-pswb"
server_key: "fa29230d068e24de99eb04190a5f77cc"
server_key_iv: "ac1b4c5a2fe96f61fe7a63e5519d6432"
server_key_version: "encryptionkey@0"
I20260620 23:16:53.273850 5122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260620 23:16:53.274745 5137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:53.274912 5122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:16:53.275018 5122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "f65162096925476fbc3616c9df814a88"
format_stamp: "Formatted at 2026-06-20 23:16:53 on dist-test-slave-pswb"
server_key: "fa29230d068e24de99eb04190a5f77cc"
server_key_iv: "ac1b4c5a2fe96f61fe7a63e5519d6432"
server_key_version: "encryptionkey@0"
I20260620 23:16:53.275105 5122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:53.289479 5122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:53.291776 5122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:53.291898 5122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:53.295786 5122 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33657
I20260620 23:16:53.295846 5189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33657 every 8 connection(s)
I20260620 23:16:53.296329 5122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:53.297003 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:53.299638 5190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88: Bootstrap starting.
I20260620 23:16:53.300256 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 5122
I20260620 23:16:53.300390 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:53.300433 5190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:53.300612 16700 external_mini_cluster.cc:1428] Setting key d00309272ca40ef4b3c12e3320755de6
I20260620 23:16:53.300778 5190 log.cc:824] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:53.301543 5190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88: No bootstrap required, opened a new log
I20260620 23:16:53.303464 5190 raft_consensus.cc:348] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65162096925476fbc3616c9df814a88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33657 } }
I20260620 23:16:53.303591 5190 raft_consensus.cc:374] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:53.303635 5190 raft_consensus.cc:729] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65162096925476fbc3616c9df814a88, State: Initialized, Role: FOLLOWER
I20260620 23:16:53.303766 5190 consensus_queue.cc:260] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65162096925476fbc3616c9df814a88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33657 } }
I20260620 23:16:53.303838 5190 raft_consensus.cc:388] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:53.303869 5190 raft_consensus.cc:482] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:53.303905 5190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:53.304610 5190 raft_consensus.cc:504] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65162096925476fbc3616c9df814a88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33657 } }
I20260620 23:16:53.304744 5190 leader_election.cc:302] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f65162096925476fbc3616c9df814a88; no voters:
I20260620 23:16:53.304911 5190 leader_election.cc:288] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:53.304947 5195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:53.305076 5195 raft_consensus.cc:686] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [term 1 LEADER]: Becoming Leader. State: Replica: f65162096925476fbc3616c9df814a88, State: Running, Role: LEADER
I20260620 23:16:53.305169 5190 sys_catalog.cc:564] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:53.305158 5195 consensus_queue.cc:237] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65162096925476fbc3616c9df814a88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33657 } }
I20260620 23:16:53.305480 5197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f65162096925476fbc3616c9df814a88. Latest consensus state: current_term: 1 leader_uuid: "f65162096925476fbc3616c9df814a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65162096925476fbc3616c9df814a88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33657 } } }
I20260620 23:16:53.305487 5196 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f65162096925476fbc3616c9df814a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65162096925476fbc3616c9df814a88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33657 } } }
I20260620 23:16:53.305637 5197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:53.305655 5196 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:53.306128 5204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:53.306643 5204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:16:53 dist-test-slave-pswb krb5kdc[5109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:16:53.308249 5204 catalog_manager.cc:1347] Generated new cluster ID: 853f816174ce43edafff30a22c0282f4
I20260620 23:16:53.308298 5204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:53.311731 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:53.301170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:59745 (local address 127.0.0.1:33657)
0620 23:16:53.301359 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:53.301369 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:53.301402 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:16:53.301882 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:53.301889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:53.301936 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:53.302075 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:53.302684 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:53.303545 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:53.304678 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:53.305326 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:53.307232 (+ 1906us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:53.308178 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:53.308181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:53.308182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:53.308192 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:53.310110 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:53.310555 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:53.310557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:53.310558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:53.310649 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:53.310892 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:53.310893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:53.310893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:53.311153 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:53.311303 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:53.311440 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:16:53.311530 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260620 23:16:53.313026 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:53.315829 5204 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:53.316480 5204 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:53.323165 5204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f65162096925476fbc3616c9df814a88: Generated new TSK 0
I20260620 23:16:53.323345 5204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:53.323594 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:53.316980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:57463 (local address 127.0.0.1:33657)
0620 23:16:53.317018 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:53.317020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:53.317158 (+ 138us) server_negotiation.cc:407] Connection header received
0620 23:16:53.317225 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:53.317226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:53.317236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:53.317387 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:53.317942 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:53.318504 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:53.319297 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:53.319535 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:53.319629 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:53.320870 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:53.320872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:53.320872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:53.320879 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:53.322305 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:53.322683 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:53.322685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:53.322685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:53.322747 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:53.322991 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:53.322992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:53.322993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:53.323122 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:53.323229 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:53.323445 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 23:16:53.323509 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:16:53.325874 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 5122
W20260620 23:16:53.330529 5107 connection.cc:535] client connection to 127.0.0.1:33657 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33657): Connection reset by peer (error 104)
2026-06-20T23:16:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:53 dist-test-slave-pswb krb5kdc[5221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:53 dist-test-slave-pswb krb5kdc[5221](info): set up 2 sockets
Jun 20 23:16:53 dist-test-slave-pswb krb5kdc[5221](info): commencing operation
krb5kdc: starting...
W20260620 23:16:54.377050 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:54 dist-test-slave-pswb krb5kdc[5221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:54.511333 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37881
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:54.590817 5234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:54.591037 5234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:54.591073 5234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:54.592166 5234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:54.592231 5234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:54.592255 5234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:54.592278 5234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:54.592305 5234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:54.593745 5234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:54.593997 5234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:54.594204 5234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:54.599488 5239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:54.599489 5240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:54.599555 5234 server_base.cc:1034] running on GCE node
W20260620 23:16:54.599467 5242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:54.599958 5234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:54.600234 5234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:54.601414 5234 hybrid_clock.cc:648] HybridClock initialized: now 1781997414601400 us; error 46 us; skew 500 ppm
Jun 20 23:16:54 dist-test-slave-pswb krb5kdc[5221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:54.603839 5234 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:54.604259 5234 webserver.cc:460] Webserver started at http://127.0.0.1:41489/ using document root <none> and password file <none>
I20260620 23:16:54.604480 5234 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:54.604545 5234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:54.604724 5234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:54.605808 5234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "8601d70d364f4bb1a6c97536da14fd22"
format_stamp: "Formatted at 2026-06-20 23:16:54 on dist-test-slave-pswb"
server_key: "52a1857bb89c9f028cd4d3ed47240eff"
server_key_iv: "0e69571b81a1ece22cf9d8224e8edefc"
server_key_version: "encryptionkey@0"
I20260620 23:16:54.606155 5234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "8601d70d364f4bb1a6c97536da14fd22"
format_stamp: "Formatted at 2026-06-20 23:16:54 on dist-test-slave-pswb"
server_key: "52a1857bb89c9f028cd4d3ed47240eff"
server_key_iv: "0e69571b81a1ece22cf9d8224e8edefc"
server_key_version: "encryptionkey@0"
I20260620 23:16:54.607571 5234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:16:54.608410 5249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:54.608606 5234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:54.608721 5234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "8601d70d364f4bb1a6c97536da14fd22"
format_stamp: "Formatted at 2026-06-20 23:16:54 on dist-test-slave-pswb"
server_key: "52a1857bb89c9f028cd4d3ed47240eff"
server_key_iv: "0e69571b81a1ece22cf9d8224e8edefc"
server_key_version: "encryptionkey@0"
I20260620 23:16:54.608799 5234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:54.626621 5234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:54.628679 5234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:54.628918 5234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:54.632943 5234 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37881
I20260620 23:16:54.632982 5301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37881 every 8 connection(s)
I20260620 23:16:54.633337 5234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:54.634196 5302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:54.636000 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 5234
I20260620 23:16:54.636107 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:54.636360 16700 external_mini_cluster.cc:1428] Setting key 788baf5192b6b528a6fef9c76d0e24d5
I20260620 23:16:54.636750 5302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22: Bootstrap starting.
I20260620 23:16:54.637383 5302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:54.637655 5302 log.cc:824] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:54.638279 5302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22: No bootstrap required, opened a new log
I20260620 23:16:54.639952 5302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8601d70d364f4bb1a6c97536da14fd22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37881 } }
I20260620 23:16:54.640075 5302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:54.640096 5302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8601d70d364f4bb1a6c97536da14fd22, State: Initialized, Role: FOLLOWER
I20260620 23:16:54.640244 5302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8601d70d364f4bb1a6c97536da14fd22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37881 } }
I20260620 23:16:54.640324 5302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:54.640362 5302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:54.640432 5302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:54.641067 5302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8601d70d364f4bb1a6c97536da14fd22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37881 } }
I20260620 23:16:54.641150 5302 leader_election.cc:302] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8601d70d364f4bb1a6c97536da14fd22; no voters:
I20260620 23:16:54.641333 5302 leader_election.cc:288] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:54.641355 5307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:54.641461 5307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [term 1 LEADER]: Becoming Leader. State: Replica: 8601d70d364f4bb1a6c97536da14fd22, State: Running, Role: LEADER
I20260620 23:16:54.641547 5307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8601d70d364f4bb1a6c97536da14fd22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37881 } }
I20260620 23:16:54.641575 5302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:54.641636 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:54.636916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:34635 (local address 127.0.0.1:37881)
0620 23:16:54.637077 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:54.637086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:54.637122 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:16:54.637589 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:54.637595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:54.637641 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:54.637796 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:54.638024 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:54.638943 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:54.639707 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:54.640128 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:54.640260 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:54.640928 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:54.640931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:54.640932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:54.641093 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:54.641141 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:54.641157 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:54.641239 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:54.641368 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 23:16:54.641397 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260620 23:16:54.641839 5307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8601d70d364f4bb1a6c97536da14fd22. Latest consensus state: current_term: 1 leader_uuid: "8601d70d364f4bb1a6c97536da14fd22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8601d70d364f4bb1a6c97536da14fd22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37881 } } }
I20260620 23:16:54.641906 5307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:54.642246 5308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8601d70d364f4bb1a6c97536da14fd22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8601d70d364f4bb1a6c97536da14fd22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37881 } } }
I20260620 23:16:54.642324 5308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:54.642652 5311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:54.642654 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:54.643122 5311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:54.644590 5311 catalog_manager.cc:1347] Generated new cluster ID: d58a3a2ab1374389a824b541721db073
I20260620 23:16:54.644644 5311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:54.651986 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:54.644127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:60143 (local address 127.0.0.1:37881)
0620 23:16:54.644158 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:54.644161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:54.648904 (+ 4743us) server_negotiation.cc:407] Connection header received
0620 23:16:54.648968 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:54.648969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:54.648987 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:54.649036 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:54.649185 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:54.649662 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:54.650350 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:54.650571 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:54.650652 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:54.651285 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:54.651286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:54.651286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:54.651291 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:54.651332 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:54.651337 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:54.651479 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:54.651822 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 23:16:54.651847 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 23:16:54.661474 5311 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:54.662163 5311 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:54.672998 5311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8601d70d364f4bb1a6c97536da14fd22: Generated new TSK 0
I20260620 23:16:54.673163 5311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:54.679754 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 5234
W20260620 23:16:54.683568 5219 connection.cc:535] client connection to 127.0.0.1:37881 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37881): Connection reset by peer (error 104)
2026-06-20T23:16:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:54 dist-test-slave-pswb krb5kdc[5333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:54 dist-test-slave-pswb krb5kdc[5333](info): set up 2 sockets
Jun 20 23:16:54 dist-test-slave-pswb krb5kdc[5333](info): commencing operation
krb5kdc: starting...
W20260620 23:16:55.728941 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:55 dist-test-slave-pswb krb5kdc[5333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:55.851756 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:55.930099 5346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:55.930342 5346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:55.930380 5346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:55.931483 5346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:55.931586 5346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:55.931612 5346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:55.931640 5346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:55.931692 5346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:55.933135 5346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:55.933398 5346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:55.933611 5346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:55.938249 5352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:55.938311 5354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:55.938266 5351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:55.938669 5346 server_base.cc:1034] running on GCE node
I20260620 23:16:55.938879 5346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:55.939139 5346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:55.940311 5346 hybrid_clock.cc:648] HybridClock initialized: now 1781997415940285 us; error 44 us; skew 500 ppm
Jun 20 23:16:55 dist-test-slave-pswb krb5kdc[5333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:55.942778 5346 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:55.943547 5346 webserver.cc:460] Webserver started at http://127.0.0.1:36777/ using document root <none> and password file <none>
I20260620 23:16:55.944017 5346 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:55.944089 5346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:55.944224 5346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:55.945806 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "fd638b21081d46fd972a4820ab9832f5"
format_stamp: "Formatted at 2026-06-20 23:16:55 on dist-test-slave-pswb"
server_key: "1f3864f658c0dee361fe485872740ca0"
server_key_iv: "e0eefd42c7c52f3c86c564f17804b471"
server_key_version: "encryptionkey@0"
I20260620 23:16:55.946290 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "fd638b21081d46fd972a4820ab9832f5"
format_stamp: "Formatted at 2026-06-20 23:16:55 on dist-test-slave-pswb"
server_key: "1f3864f658c0dee361fe485872740ca0"
server_key_iv: "e0eefd42c7c52f3c86c564f17804b471"
server_key_version: "encryptionkey@0"
I20260620 23:16:55.948046 5346 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:16:55.949014 5361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:55.949256 5346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:55.949365 5346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "fd638b21081d46fd972a4820ab9832f5"
format_stamp: "Formatted at 2026-06-20 23:16:55 on dist-test-slave-pswb"
server_key: "1f3864f658c0dee361fe485872740ca0"
server_key_iv: "e0eefd42c7c52f3c86c564f17804b471"
server_key_version: "encryptionkey@0"
I20260620 23:16:55.949470 5346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:55.952768 5346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:55.954891 5346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:55.955051 5346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:55.959057 5346 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46187
I20260620 23:16:55.959085 5413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46187 every 8 connection(s)
I20260620 23:16:55.959407 5346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:55.960194 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:55.962402 5414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5: Bootstrap starting.
I20260620 23:16:55.962904 5414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:55.963124 5414 log.cc:824] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:55.963950 5414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5: No bootstrap required, opened a new log
I20260620 23:16:55.965765 5414 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd638b21081d46fd972a4820ab9832f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46187 } }
I20260620 23:16:55.965920 5414 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:55.965946 5414 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd638b21081d46fd972a4820ab9832f5, State: Initialized, Role: FOLLOWER
I20260620 23:16:55.966092 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 5346
I20260620 23:16:55.966099 5414 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd638b21081d46fd972a4820ab9832f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46187 } }
I20260620 23:16:55.966154 5414 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:55.966202 5414 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:55.966217 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:55.966254 5414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:55.966545 16700 external_mini_cluster.cc:1428] Setting key 35124edc72eaf4c94bd46272585e268a
I20260620 23:16:55.966892 5414 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd638b21081d46fd972a4820ab9832f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46187 } }
I20260620 23:16:55.967015 5414 leader_election.cc:302] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd638b21081d46fd972a4820ab9832f5; no voters:
I20260620 23:16:55.967250 5414 leader_election.cc:288] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:55.967334 5418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:55.967571 5414 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:55.967684 5418 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [term 1 LEADER]: Becoming Leader. State: Replica: fd638b21081d46fd972a4820ab9832f5, State: Running, Role: LEADER
I20260620 23:16:55.967855 5418 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd638b21081d46fd972a4820ab9832f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46187 } }
I20260620 23:16:55.968472 5418 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd638b21081d46fd972a4820ab9832f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd638b21081d46fd972a4820ab9832f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46187 } } }
I20260620 23:16:55.968516 5421 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd638b21081d46fd972a4820ab9832f5. Latest consensus state: current_term: 1 leader_uuid: "fd638b21081d46fd972a4820ab9832f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd638b21081d46fd972a4820ab9832f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46187 } } }
I20260620 23:16:55.968737 5421 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:55.968607 5418 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:55.968976 5428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:55.969403 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:55.967225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:49603 (local address 127.0.0.1:46187)
0620 23:16:55.967405 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:55.967414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:55.967477 (+ 63us) server_negotiation.cc:407] Connection header received
0620 23:16:55.967591 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:55.967598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:55.967627 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:55.967775 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:55.968791 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:55.968795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:55.968795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:55.968802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:55.968852 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:55.968866 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:55.968952 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:55.969150 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 23:16:55.969203 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":61,"threads_started":1}
I20260620 23:16:55.969535 5428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:55.970449 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:55.971235 5428 catalog_manager.cc:1347] Generated new cluster ID: 9c505de0e39e42ad87318ada0dadfbc5
I20260620 23:16:55.971298 5428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:55.977787 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:55.972957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:49635 (local address 127.0.0.1:46187)
0620 23:16:55.973001 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:55.973003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:55.976845 (+ 3842us) server_negotiation.cc:407] Connection header received
0620 23:16:55.976897 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:55.976898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:55.976903 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:55.976965 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:55.977560 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:55.977561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:55.977561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:55.977565 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:55.977581 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:55.977586 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:55.977648 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:55.977705 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 23:16:55.977721 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:16:56.000289 5428 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:56.000468 5428 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:56.005148 5428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd638b21081d46fd972a4820ab9832f5: Generated new TSK 0
I20260620 23:16:56.005324 5428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:56.010010 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 5346
2026-06-20T23:16:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:56 dist-test-slave-pswb krb5kdc[5445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:56 dist-test-slave-pswb krb5kdc[5445](info): set up 2 sockets
Jun 20 23:16:56 dist-test-slave-pswb krb5kdc[5445](info): commencing operation
krb5kdc: starting...
W20260620 23:16:57.093148 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:57 dist-test-slave-pswb krb5kdc[5445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:57.215232 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45589
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:57.294473 5458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:57.294648 5458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:57.294677 5458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:57.295714 5458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:57.295750 5458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:57.295764 5458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:57.295776 5458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:57.295789 5458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:57.297140 5458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45589
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:57.297307 5458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:57.297461 5458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:57.302145 5464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:57.302290 5463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:57.302141 5466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:57.302525 5458 server_base.cc:1034] running on GCE node
I20260620 23:16:57.302872 5458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:57.303197 5458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:57.304353 5458 hybrid_clock.cc:648] HybridClock initialized: now 1781997417304340 us; error 30 us; skew 500 ppm
Jun 20 23:16:57 dist-test-slave-pswb krb5kdc[5445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:57.307340 5458 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:57.307721 5458 webserver.cc:460] Webserver started at http://127.0.0.1:41345/ using document root <none> and password file <none>
I20260620 23:16:57.307909 5458 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:57.307950 5458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:57.308037 5458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:57.309063 5458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "5e10ba19a43847b2af5bef5830114207"
format_stamp: "Formatted at 2026-06-20 23:16:57 on dist-test-slave-pswb"
server_key: "cebcf5c16f08fdd727ca59ee266483fe"
server_key_iv: "44846f0fcbf90735783f54087aada7e9"
server_key_version: "encryptionkey@0"
I20260620 23:16:57.309422 5458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "5e10ba19a43847b2af5bef5830114207"
format_stamp: "Formatted at 2026-06-20 23:16:57 on dist-test-slave-pswb"
server_key: "cebcf5c16f08fdd727ca59ee266483fe"
server_key_iv: "44846f0fcbf90735783f54087aada7e9"
server_key_version: "encryptionkey@0"
I20260620 23:16:57.310662 5458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 23:16:57.311424 5473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:57.311641 5458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:16:57.311741 5458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "5e10ba19a43847b2af5bef5830114207"
format_stamp: "Formatted at 2026-06-20 23:16:57 on dist-test-slave-pswb"
server_key: "cebcf5c16f08fdd727ca59ee266483fe"
server_key_iv: "44846f0fcbf90735783f54087aada7e9"
server_key_version: "encryptionkey@0"
I20260620 23:16:57.311812 5458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:57.331779 5458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:57.334158 5458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:57.334267 5458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:57.338054 5458 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45589
I20260620 23:16:57.338099 5525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45589 every 8 connection(s)
I20260620 23:16:57.338434 5458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:57.339264 5526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:57.339754 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 5458
I20260620 23:16:57.339882 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:57.340080 16700 external_mini_cluster.cc:1428] Setting key e496dfeb4522d7fd0de073c40c4ea9d4
I20260620 23:16:57.342149 5526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207: Bootstrap starting.
I20260620 23:16:57.342730 5526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:57.343017 5526 log.cc:824] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:57.343770 5526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207: No bootstrap required, opened a new log
I20260620 23:16:57.346287 5526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e10ba19a43847b2af5bef5830114207" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45589 } }
I20260620 23:16:57.346418 5526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:57.346441 5526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e10ba19a43847b2af5bef5830114207, State: Initialized, Role: FOLLOWER
I20260620 23:16:57.346550 5526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e10ba19a43847b2af5bef5830114207" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45589 } }
I20260620 23:16:57.346625 5526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:57.346653 5526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:57.346701 5526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 23:16:57 dist-test-slave-pswb krb5kdc[5445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:16:57.347267 5526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e10ba19a43847b2af5bef5830114207" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45589 } }
I20260620 23:16:57.347388 5526 leader_election.cc:302] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e10ba19a43847b2af5bef5830114207; no voters:
I20260620 23:16:57.347555 5526 leader_election.cc:288] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:57.347615 5531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:57.347801 5526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:57.348049 5531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [term 1 LEADER]: Becoming Leader. State: Replica: 5e10ba19a43847b2af5bef5830114207, State: Running, Role: LEADER
I20260620 23:16:57.348160 5531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e10ba19a43847b2af5bef5830114207" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45589 } }
I20260620 23:16:57.348973 5532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e10ba19a43847b2af5bef5830114207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e10ba19a43847b2af5bef5830114207" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45589 } } }
I20260620 23:16:57.349049 5532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:57.349380 5533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e10ba19a43847b2af5bef5830114207. Latest consensus state: current_term: 1 leader_uuid: "5e10ba19a43847b2af5bef5830114207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e10ba19a43847b2af5bef5830114207" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45589 } } }
W20260620 23:16:57.349812 5548 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 23:16:57.350186 5533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207 [sys.catalog]: This master's current role is: LEADER
W20260620 23:16:57.350198 5548 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:16:57.350293 5543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:57.350514 5543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:57.351598 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:57.340592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:41449 (local address 127.0.0.1:45589)
0620 23:16:57.340767 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:57.340774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:57.340798 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:16:57.341186 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:57.341192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:57.341237 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:57.341347 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:57.342138 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:57.343076 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:57.344081 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:57.344464 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:57.344556 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:57.347393 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:57.347397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:57.347398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:57.347409 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:57.349562 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:57.350022 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:57.350026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:57.350026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:57.350139 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:57.350421 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:57.350422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:57.350422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:57.350710 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:57.350842 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:57.351295 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 23:16:57.351384 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260620 23:16:57.352257 5543 catalog_manager.cc:1347] Generated new cluster ID: b35ab052a17c4bdeb9e6d87ffeffe943
I20260620 23:16:57.352305 5543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:57.352555 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:57.360852 5543 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:57.361073 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:57.353903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:43653 (local address 127.0.0.1:45589)
0620 23:16:57.353937 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:57.353939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:57.354063 (+ 124us) server_negotiation.cc:407] Connection header received
0620 23:16:57.354127 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:57.354127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:57.354137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:57.354194 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:57.354730 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:57.355514 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:57.356468 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:57.356838 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:57.356918 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:57.358164 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:57.358165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:57.358165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:57.358173 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:57.359722 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:57.360133 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:57.360134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:57.360134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:57.360194 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:57.360421 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:57.360422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:57.360423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:57.360546 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:57.360677 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:57.360924 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 23:16:57.360987 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:16:57.361469 5543 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:57.367646 5543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e10ba19a43847b2af5bef5830114207: Generated new TSK 0
I20260620 23:16:57.367823 5543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:57.371348 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 5458
W20260620 23:16:57.375686 5443 connection.cc:535] client connection to 127.0.0.1:45589 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45589): Connection reset by peer (error 104)
2026-06-20T23:16:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:57 dist-test-slave-pswb krb5kdc[5558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:57 dist-test-slave-pswb krb5kdc[5558](info): set up 2 sockets
Jun 20 23:16:57 dist-test-slave-pswb krb5kdc[5558](info): commencing operation
krb5kdc: starting...
W20260620 23:16:58.417572 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:58 dist-test-slave-pswb krb5kdc[5558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:58.533793 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:58.615368 5571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:58.615581 5571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:58.615615 5571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:58.616724 5571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:58.616784 5571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:58.616809 5571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:58.616854 5571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:58.616882 5571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:58.618367 5571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:58.618624 5571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:58.618893 5571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:58.623680 5577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:58.623689 5579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:58.623679 5576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:58.623914 5571 server_base.cc:1034] running on GCE node
I20260620 23:16:58.624126 5571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:58.624347 5571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:58.625510 5571 hybrid_clock.cc:648] HybridClock initialized: now 1781997418625497 us; error 27 us; skew 500 ppm
Jun 20 23:16:58 dist-test-slave-pswb krb5kdc[5558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:58.627875 5571 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:58.628296 5571 webserver.cc:460] Webserver started at http://127.0.0.1:38227/ using document root <none> and password file <none>
I20260620 23:16:58.628530 5571 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:58.628593 5571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:58.628767 5571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:58.629760 5571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "97ee4b867b504f09ba0d75f2fbb2cf76"
format_stamp: "Formatted at 2026-06-20 23:16:58 on dist-test-slave-pswb"
server_key: "d76ecf87ed8748713c1a1a0562ec1bc2"
server_key_iv: "1fbf6059c706ed8ef3ea1fdd5f3f13fd"
server_key_version: "encryptionkey@0"
I20260620 23:16:58.630106 5571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "97ee4b867b504f09ba0d75f2fbb2cf76"
format_stamp: "Formatted at 2026-06-20 23:16:58 on dist-test-slave-pswb"
server_key: "d76ecf87ed8748713c1a1a0562ec1bc2"
server_key_iv: "1fbf6059c706ed8ef3ea1fdd5f3f13fd"
server_key_version: "encryptionkey@0"
I20260620 23:16:58.631382 5571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:16:58.632036 5586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:58.632207 5571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:16:58.632296 5571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "97ee4b867b504f09ba0d75f2fbb2cf76"
format_stamp: "Formatted at 2026-06-20 23:16:58 on dist-test-slave-pswb"
server_key: "d76ecf87ed8748713c1a1a0562ec1bc2"
server_key_iv: "1fbf6059c706ed8ef3ea1fdd5f3f13fd"
server_key_version: "encryptionkey@0"
I20260620 23:16:58.632375 5571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:58.647924 5571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:58.649813 5571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:58.649960 5571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:58.653729 5571 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34893
I20260620 23:16:58.653760 5638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34893 every 8 connection(s)
I20260620 23:16:58.654071 5571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:58.654901 5639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:58.657022 5639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76: Bootstrap starting.
I20260620 23:16:58.657553 5639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:58.657820 5639 log.cc:824] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:58.658443 5639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76: No bootstrap required, opened a new log
I20260620 23:16:58.659440 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 5571
I20260620 23:16:58.659574 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:58.659754 16700 external_mini_cluster.cc:1428] Setting key fd44e5adc7ad625b1630302f48c631e8
I20260620 23:16:58.660168 5639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ee4b867b504f09ba0d75f2fbb2cf76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34893 } }
I20260620 23:16:58.660372 5639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:58.660425 5639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ee4b867b504f09ba0d75f2fbb2cf76, State: Initialized, Role: FOLLOWER
I20260620 23:16:58.660552 5639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ee4b867b504f09ba0d75f2fbb2cf76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34893 } }
I20260620 23:16:58.660624 5639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:58.660661 5639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:58.660725 5639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:58.661645 5639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ee4b867b504f09ba0d75f2fbb2cf76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34893 } }
I20260620 23:16:58.661729 5639 leader_election.cc:302] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97ee4b867b504f09ba0d75f2fbb2cf76; no voters:
I20260620 23:16:58.661885 5639 leader_election.cc:288] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:58.661911 5644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:58.662014 5644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [term 1 LEADER]: Becoming Leader. State: Replica: 97ee4b867b504f09ba0d75f2fbb2cf76, State: Running, Role: LEADER
I20260620 23:16:58.662123 5639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:58.662146 5644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ee4b867b504f09ba0d75f2fbb2cf76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34893 } }
I20260620 23:16:58.662508 5644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97ee4b867b504f09ba0d75f2fbb2cf76. Latest consensus state: current_term: 1 leader_uuid: "97ee4b867b504f09ba0d75f2fbb2cf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ee4b867b504f09ba0d75f2fbb2cf76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34893 } } }
I20260620 23:16:58.662765 5644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:58.662672 5645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97ee4b867b504f09ba0d75f2fbb2cf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ee4b867b504f09ba0d75f2fbb2cf76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34893 } } }
I20260620 23:16:58.662909 5645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:58.663149 5653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:58.663695 5653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:58.665427 5653 catalog_manager.cc:1347] Generated new cluster ID: 78e89ab4acd64d13aae3256b9500a802
I20260620 23:16:58.665445 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:58.660263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:58929 (local address 127.0.0.1:34893)
0620 23:16:58.660478 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:58.660489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:58.660536 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:16:58.660670 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:58.660676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:58.660767 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:58.660889 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:58.661145 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:58.662172 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:58.663269 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:58.663865 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:58.664254 (+ 389us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:58.664675 (+ 421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:58.664678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:58.664678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:58.664899 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:58.664933 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:58.664947 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:58.665027 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:58.665194 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 23:16:58.665221 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20260620 23:16:58.665491 5653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:58.666253 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:58.670225 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:58.667495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:35083 (local address 127.0.0.1:34893)
0620 23:16:58.667546 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:58.667548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:58.667658 (+ 110us) server_negotiation.cc:407] Connection header received
0620 23:16:58.667727 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:58.667728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:58.667737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:58.667793 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:58.667923 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:58.668361 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:58.669106 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:58.669314 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:58.669398 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:58.669994 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:58.669995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:58.669995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:58.670000 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:58.670012 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:58.670016 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:58.670068 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:58.670126 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 23:16:58.670136 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:16:58.676012 5653 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:58.676484 5653 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:58.679528 5653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97ee4b867b504f09ba0d75f2fbb2cf76: Generated new TSK 0
I20260620 23:16:58.679674 5653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:58.687850 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 5571
W20260620 23:16:58.693193 5556 connection.cc:535] client connection to 127.0.0.1:34893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34893): Connection reset by peer (error 104)
2026-06-20T23:16:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:58 dist-test-slave-pswb krb5kdc[5670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:58 dist-test-slave-pswb krb5kdc[5670](info): set up 2 sockets
Jun 20 23:16:58 dist-test-slave-pswb krb5kdc[5670](info): commencing operation
krb5kdc: starting...
W20260620 23:16:59.733103 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:16:59 dist-test-slave-pswb krb5kdc[5670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:59.855448 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:59.934093 5683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:59.934325 5683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:59.934361 5683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:59.935477 5683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:59.935537 5683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:59.935560 5683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:59.935599 5683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:59.935627 5683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:59.937079 5683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43137
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:16:59.937330 5683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:59.937530 5683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:59.942365 5691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:59.942365 5689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:59.942551 5683 server_base.cc:1034] running on GCE node
W20260620 23:16:59.942369 5688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:59.942883 5683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:59.943130 5683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:59.944285 5683 hybrid_clock.cc:648] HybridClock initialized: now 1781997419944238 us; error 72 us; skew 500 ppm
Jun 20 23:16:59 dist-test-slave-pswb krb5kdc[5670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:59.951576 5683 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:59.952030 5683 webserver.cc:460] Webserver started at http://127.0.0.1:33645/ using document root <none> and password file <none>
I20260620 23:16:59.952237 5683 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:59.952280 5683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:59.952451 5683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:59.953552 5683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "3cfb4dae6751439da3f5ff88cacda378"
format_stamp: "Formatted at 2026-06-20 23:16:59 on dist-test-slave-pswb"
server_key: "09695fd3784881d911e41788b459b5e0"
server_key_iv: "c9ab92e3a767b4978c7e47935c75d3c0"
server_key_version: "encryptionkey@0"
I20260620 23:16:59.953902 5683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "3cfb4dae6751439da3f5ff88cacda378"
format_stamp: "Formatted at 2026-06-20 23:16:59 on dist-test-slave-pswb"
server_key: "09695fd3784881d911e41788b459b5e0"
server_key_iv: "c9ab92e3a767b4978c7e47935c75d3c0"
server_key_version: "encryptionkey@0"
I20260620 23:16:59.955197 5683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:16:59.955938 5698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:59.956097 5683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:59.956190 5683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "3cfb4dae6751439da3f5ff88cacda378"
format_stamp: "Formatted at 2026-06-20 23:16:59 on dist-test-slave-pswb"
server_key: "09695fd3784881d911e41788b459b5e0"
server_key_iv: "c9ab92e3a767b4978c7e47935c75d3c0"
server_key_version: "encryptionkey@0"
I20260620 23:16:59.956270 5683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:59.959474 5683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:59.961653 5683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:59.961795 5683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:59.966239 5683 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43137
I20260620 23:16:59.966374 5750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43137 every 8 connection(s)
I20260620 23:16:59.966626 5683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:59.967671 5751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:59.970060 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 5683
I20260620 23:16:59.970160 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:16:59.970366 16700 external_mini_cluster.cc:1428] Setting key 234375f95262abf33bce3da29e739fca
I20260620 23:16:59.970652 5751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378: Bootstrap starting.
I20260620 23:16:59.972023 5751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:59.972357 5751 log.cc:824] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:59.973277 5751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378: No bootstrap required, opened a new log
I20260620 23:16:59.973469 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:59.970974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:51559 (local address 127.0.0.1:43137)
0620 23:16:59.971282 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:59.971293 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:59.971335 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:16:59.972081 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:59.972090 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:59.972123 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:59.972295 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:59.973040 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:59.973043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:59.973043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:59.973047 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:59.973082 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:59.973092 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:59.973167 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:59.973228 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 23:16:59.973280 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":171,"threads_started":1}
I20260620 23:16:59.974413 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:59.975363 5751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cfb4dae6751439da3f5ff88cacda378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43137 } }
I20260620 23:16:59.975502 5751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:59.975545 5751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cfb4dae6751439da3f5ff88cacda378, State: Initialized, Role: FOLLOWER
I20260620 23:16:59.975660 5751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cfb4dae6751439da3f5ff88cacda378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43137 } }
I20260620 23:16:59.975744 5751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:59.975785 5751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:59.975832 5751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:59.976336 5751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cfb4dae6751439da3f5ff88cacda378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43137 } }
I20260620 23:16:59.976444 5751 leader_election.cc:302] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cfb4dae6751439da3f5ff88cacda378; no voters:
I20260620 23:16:59.976629 5751 leader_election.cc:288] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:59.976650 5761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:59.976811 5761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [term 1 LEADER]: Becoming Leader. State: Replica: 3cfb4dae6751439da3f5ff88cacda378, State: Running, Role: LEADER
I20260620 23:16:59.976922 5751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:59.977023 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:59.975844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:41405 (local address 127.0.0.1:43137)
0620 23:16:59.975876 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:59.975878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:59.976086 (+ 208us) server_negotiation.cc:407] Connection header received
0620 23:16:59.976143 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:59.976144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:59.976149 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:59.976203 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:59.976764 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:59.976765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:59.976765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:59.976769 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:59.976782 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:59.976786 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:59.976817 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:59.976922 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 23:16:59.976934 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 23:16:59.976940 5761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cfb4dae6751439da3f5ff88cacda378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43137 } }
I20260620 23:16:59.977586 5761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cfb4dae6751439da3f5ff88cacda378. Latest consensus state: current_term: 1 leader_uuid: "3cfb4dae6751439da3f5ff88cacda378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cfb4dae6751439da3f5ff88cacda378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43137 } } }
I20260620 23:16:59.977698 5761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:59.977597 5763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cfb4dae6751439da3f5ff88cacda378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cfb4dae6751439da3f5ff88cacda378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43137 } } }
I20260620 23:16:59.977857 5763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:59.977906 5770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:59.978384 5770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:59.979832 5770 catalog_manager.cc:1347] Generated new cluster ID: 4b079c6d376c460ea37237d18bb06ea5
I20260620 23:16:59.979869 5770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:59.996474 5770 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:59.996636 5770 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:17:00.012825 5770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cfb4dae6751439da3f5ff88cacda378: Generated new TSK 0
I20260620 23:17:00.013028 5770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:17:00.019459 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 5683
2026-06-20T23:17:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:17:00 dist-test-slave-pswb krb5kdc[5782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:17:00 dist-test-slave-pswb krb5kdc[5782](info): set up 2 sockets
Jun 20 23:17:00 dist-test-slave-pswb krb5kdc[5782](info): commencing operation
krb5kdc: starting...
W20260620 23:17:01.088907 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:17:01 dist-test-slave-pswb krb5kdc[5782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:17:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:17:01.208817 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:37805
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:17:01.289608 5795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:17:01.289831 5795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:17:01.289870 5795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:17:01.290994 5795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:17:01.291056 5795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:17:01.291080 5795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:17:01.291107 5795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:17:01.291139 5795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:17:01.292608 5795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:37805
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:37805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:17:01.292896 5795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:17:01.293104 5795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:17:01.297863 5800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:01.298198 5801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:01.298522 5803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:01.299984 5795 server_base.cc:1034] running on GCE node
I20260620 23:17:01.300288 5795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:17:01.300644 5795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:17:01.304150 5795 hybrid_clock.cc:648] HybridClock initialized: now 1781997421304133 us; error 42 us; skew 500 ppm
Jun 20 23:17:01 dist-test-slave-pswb krb5kdc[5782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997421, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:17:01.307526 5795 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:17:01.308132 5795 webserver.cc:460] Webserver started at http://127.16.79.62:33715/ using document root <none> and password file <none>
I20260620 23:17:01.308429 5795 fs_manager.cc:359] Metadata directory not provided
I20260620 23:17:01.308495 5795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:17:01.308669 5795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:17:01.310055 5795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "5dd14f70256e4e2ba126171604da3bdf"
format_stamp: "Formatted at 2026-06-20 23:17:01 on dist-test-slave-pswb"
server_key: "cdd8ec3f4571cab5fbb23023b36d2a24"
server_key_iv: "439c35483d6f240c1c0cb0941d3d44c8"
server_key_version: "encryptionkey@0"
I20260620 23:17:01.310415 5795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "5dd14f70256e4e2ba126171604da3bdf"
format_stamp: "Formatted at 2026-06-20 23:17:01 on dist-test-slave-pswb"
server_key: "cdd8ec3f4571cab5fbb23023b36d2a24"
server_key_iv: "439c35483d6f240c1c0cb0941d3d44c8"
server_key_version: "encryptionkey@0"
I20260620 23:17:01.312103 5795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:17:01.313139 5810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:17:01.313304 5795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:17:01.313397 5795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "5dd14f70256e4e2ba126171604da3bdf"
format_stamp: "Formatted at 2026-06-20 23:17:01 on dist-test-slave-pswb"
server_key: "cdd8ec3f4571cab5fbb23023b36d2a24"
server_key_iv: "439c35483d6f240c1c0cb0941d3d44c8"
server_key_version: "encryptionkey@0"
I20260620 23:17:01.313454 5795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:17:01.333323 5795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:17:01.335568 5795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:17:01.335683 5795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:17:01.339815 5795 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:37805
I20260620 23:17:01.339848 5862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:37805 every 8 connection(s)
I20260620 23:17:01.340191 5795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:17:01.341117 5863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:17:01.343235 5863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf: Bootstrap starting.
I20260620 23:17:01.343787 5863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf: Neither blocks nor log segments found. Creating new log.
I20260620 23:17:01.344058 5863 log.cc:824] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf: Log is configured to *not* fsync() on all Append() calls
I20260620 23:17:01.344259 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 5795
I20260620 23:17:01.344409 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:17:01.344601 16700 external_mini_cluster.cc:1428] Setting key e7f2c6156f5be09fd1981a099947000e
I20260620 23:17:01.345173 5863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf: No bootstrap required, opened a new log
I20260620 23:17:01.347604 5863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd14f70256e4e2ba126171604da3bdf" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 37805 } }
I20260620 23:17:01.347772 5863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:17:01.347838 5863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd14f70256e4e2ba126171604da3bdf, State: Initialized, Role: FOLLOWER
I20260620 23:17:01.348001 5863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd14f70256e4e2ba126171604da3bdf" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 37805 } }
I20260620 23:17:01.348084 5863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:17:01.348130 5863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:17:01.348179 5863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:17:01.349102 5863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd14f70256e4e2ba126171604da3bdf" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 37805 } }
I20260620 23:17:01.349224 5863 leader_election.cc:302] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dd14f70256e4e2ba126171604da3bdf; no voters:
I20260620 23:17:01.349412 5863 leader_election.cc:288] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:17:01.349447 5868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:17:01.349547 5868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [term 1 LEADER]: Becoming Leader. State: Replica: 5dd14f70256e4e2ba126171604da3bdf, State: Running, Role: LEADER
I20260620 23:17:01.349666 5863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:17:01.349675 5868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd14f70256e4e2ba126171604da3bdf" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 37805 } }
I20260620 23:17:01.350196 5870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dd14f70256e4e2ba126171604da3bdf. Latest consensus state: current_term: 1 leader_uuid: "5dd14f70256e4e2ba126171604da3bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd14f70256e4e2ba126171604da3bdf" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 37805 } } }
I20260620 23:17:01.350328 5870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:01.350531 5869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dd14f70256e4e2ba126171604da3bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd14f70256e4e2ba126171604da3bdf" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 37805 } } }
I20260620 23:17:01.350631 5869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:01.350924 5877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:17:01.351449 5877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:17:01.353012 5877 catalog_manager.cc:1347] Generated new cluster ID: 2b97a81dc1c6414a97c55fa1a407a35c
I20260620 23:17:01.353070 5877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:17:01 dist-test-slave-pswb krb5kdc[5782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:17:01.359336 5867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:01.345246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43925 (local address 127.16.79.62:37805)
0620 23:17:01.345467 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:01.345477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:01.345536 (+ 59us) server_negotiation.cc:407] Connection header received
0620 23:17:01.347599 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:01.347608 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:01.347662 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:01.347858 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:01.348614 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:01.349893 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:01.351007 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:01.351614 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:01.352020 (+ 406us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:01.354838 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:01.354842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:01.354842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:01.354854 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:01.356908 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:01.357589 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:01.357592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:01.357592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:01.357734 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:01.358107 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:01.358108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:01.358108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:01.358399 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:01.358960 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:01.359035 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 23:17:01.359135 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":111,"threads_started":1}
I20260620 23:17:01.360770 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:17:01.361294 5877 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:17:01.362010 5877 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:17:01.364912 5877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dd14f70256e4e2ba126171604da3bdf: Generated new TSK 0
I20260620 23:17:01.365069 5877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:17:01.369814 5867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:01.362343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50949 (local address 127.16.79.62:37805)
0620 23:17:01.362381 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:01.362382 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:01.362515 (+ 133us) server_negotiation.cc:407] Connection header received
0620 23:17:01.362578 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:01.362579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:01.362589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:01.362650 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:01.363235 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:01.364023 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:01.365074 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:01.365385 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:01.365425 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:01.366608 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:01.366609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:01.366610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:01.366616 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:01.368383 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:01.368830 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:01.368831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:01.368831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:01.368889 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:01.369161 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:01.369162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:01.369162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:01.369287 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:01.369386 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:01.369542 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 23:17:01.369604 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:17:01.372372 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 5795
W20260620 23:17:01.377628 5780 connection.cc:535] client connection to 127.16.79.62:37805 recv error: Network error: failed to read from TLS socket (remote: 127.16.79.62:37805): Connection reset by peer (error 104)
2026-06-20T23:17:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:17:01 dist-test-slave-pswb krb5kdc[5894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:17:01 dist-test-slave-pswb krb5kdc[5894](info): set up 2 sockets
Jun 20 23:17:01 dist-test-slave-pswb krb5kdc[5894](info): commencing operation
krb5kdc: starting...
W20260620 23:17:02.445148 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:17:02 dist-test-slave-pswb krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:17:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:17:02.568043 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36583
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:36583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:17:02.655093 5907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:17:02.655333 5907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:17:02.655370 5907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:17:02.656488 5907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:17:02.656553 5907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:17:02.656579 5907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:17:02.656607 5907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:17:02.656630 5907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:17:02.658084 5907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:36583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:17:02.658360 5907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:17:02.658571 5907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:17:02.663283 5915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:02.663295 5913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:02.663296 5912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:02.663719 5907 server_base.cc:1034] running on GCE node
I20260620 23:17:02.663965 5907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:17:02.664242 5907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:17:02.665464 5907 hybrid_clock.cc:648] HybridClock initialized: now 1781997422665462 us; error 40 us; skew 500 ppm
Jun 20 23:17:02 dist-test-slave-pswb krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997422, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:17:02.667827 5907 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:17:02.668291 5907 webserver.cc:460] Webserver started at http://127.16.79.62:38477/ using document root <none> and password file <none>
I20260620 23:17:02.668506 5907 fs_manager.cc:359] Metadata directory not provided
I20260620 23:17:02.668572 5907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:17:02.668761 5907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:17:02.669756 5907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "777711146513471e8cffab4d8be7304d"
format_stamp: "Formatted at 2026-06-20 23:17:02 on dist-test-slave-pswb"
server_key: "77cf86a2bc89b5070c3efd6847de88a1"
server_key_iv: "428985a6789348aaea80e8c2b482e4d8"
server_key_version: "encryptionkey@0"
I20260620 23:17:02.670154 5907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "777711146513471e8cffab4d8be7304d"
format_stamp: "Formatted at 2026-06-20 23:17:02 on dist-test-slave-pswb"
server_key: "77cf86a2bc89b5070c3efd6847de88a1"
server_key_iv: "428985a6789348aaea80e8c2b482e4d8"
server_key_version: "encryptionkey@0"
I20260620 23:17:02.672240 5907 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:17:02.673511 5922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:17:02.673779 5907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:17:02.673913 5907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "777711146513471e8cffab4d8be7304d"
format_stamp: "Formatted at 2026-06-20 23:17:02 on dist-test-slave-pswb"
server_key: "77cf86a2bc89b5070c3efd6847de88a1"
server_key_iv: "428985a6789348aaea80e8c2b482e4d8"
server_key_version: "encryptionkey@0"
I20260620 23:17:02.674015 5907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:17:02.686025 5907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:17:02.688050 5907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:17:02.688206 5907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:17:02.692509 5907 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:36583
I20260620 23:17:02.692548 5974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:36583 every 8 connection(s)
I20260620 23:17:02.692934 5907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:17:02.693425 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 5907
I20260620 23:17:02.693512 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:17:02.693718 16700 external_mini_cluster.cc:1428] Setting key 5de5ac8896a39f2d2614d7426df4a28b
I20260620 23:17:02.693874 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:17:02.696802 5975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d: Bootstrap starting.
I20260620 23:17:02.697477 5975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d: Neither blocks nor log segments found. Creating new log.
I20260620 23:17:02.697768 5975 log.cc:824] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:17:02.698571 5975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d: No bootstrap required, opened a new log
I20260620 23:17:02.699681 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:02.694552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39001 (local address 127.16.79.62:36583)
0620 23:17:02.694714 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:02.694723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:02.694752 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:17:02.695197 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:02.695203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:02.695250 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:02.695352 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:02.695617 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:02.696485 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:02.697607 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:02.698168 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:02.698288 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:02.699001 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:02.699004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:02.699005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:17:02.699158 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:02.699185 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:17:02.699195 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:02.699256 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:02.699452 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 23:17:02.699484 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260620 23:17:02.700852 5975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777711146513471e8cffab4d8be7304d" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36583 } }
I20260620 23:17:02.701036 5975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:17:02.701084 5975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777711146513471e8cffab4d8be7304d, State: Initialized, Role: FOLLOWER
I20260620 23:17:02.701097 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:17:02.701196 5975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777711146513471e8cffab4d8be7304d" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36583 } }
I20260620 23:17:02.701278 5975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:17:02.701320 5975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:17:02.701366 5975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:17:02.702059 5975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777711146513471e8cffab4d8be7304d" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36583 } }
I20260620 23:17:02.702283 5975 leader_election.cc:302] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 777711146513471e8cffab4d8be7304d; no voters:
I20260620 23:17:02.702548 5975 leader_election.cc:288] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:17:02.702867 5985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:17:02.702982 5975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:17:02.703094 5985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [term 1 LEADER]: Becoming Leader. State: Replica: 777711146513471e8cffab4d8be7304d, State: Running, Role: LEADER
I20260620 23:17:02.703274 5985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777711146513471e8cffab4d8be7304d" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36583 } }
I20260620 23:17:02.703823 5984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "777711146513471e8cffab4d8be7304d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777711146513471e8cffab4d8be7304d" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36583 } } }
I20260620 23:17:02.703941 5984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:02.704087 5984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 777711146513471e8cffab4d8be7304d. Latest consensus state: current_term: 1 leader_uuid: "777711146513471e8cffab4d8be7304d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777711146513471e8cffab4d8be7304d" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36583 } } }
I20260620 23:17:02.704125 5984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:02.704356 5994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:17:02.704945 5994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:17:02.706904 5994 catalog_manager.cc:1347] Generated new cluster ID: 0f46e371861749ffa765727eb4bdd8e6
I20260620 23:17:02.706974 5994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:17:02.707159 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:02.702970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50637 (local address 127.16.79.62:36583)
0620 23:17:02.703062 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:02.703064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:02.703074 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:17:02.703197 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:02.703198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:02.703210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:02.703245 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:02.703644 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:02.704385 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:02.705724 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:02.706015 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:02.706050 (+ 35us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:02.706845 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:02.706847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:02.706847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:17:02.706852 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:02.706871 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:17:02.706877 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:02.706911 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:02.707081 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 23:17:02.707095 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:17:02.716605 5994 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:17:02.717167 5994 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:17:02.726608 5994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 777711146513471e8cffab4d8be7304d: Generated new TSK 0
I20260620 23:17:02.726804 5994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:17:02.738304 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 5907
W20260620 23:17:02.742542 5892 connection.cc:535] client connection to 127.16.79.62:36583 recv error: Network error: failed to read from TLS socket (remote: 127.16.79.62:36583): Connection reset by peer (error 104)
2026-06-20T23:17:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:17:02 dist-test-slave-pswb krb5kdc[6006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:17:02 dist-test-slave-pswb krb5kdc[6006](info): set up 2 sockets
Jun 20 23:17:02 dist-test-slave-pswb krb5kdc[6006](info): commencing operation
krb5kdc: starting...
W20260620 23:17:03.788938 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:17:03 dist-test-slave-pswb krb5kdc[6006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:17:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:17:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:17:03.915998 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:43757
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:39111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:43757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:17:03.996582 6019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:17:03.996863 6019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:17:03.996906 6019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:17:03.998003 6019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:17:03.998067 6019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:17:03.998092 6019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:17:03.998145 6019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:17:03.998167 6019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:17:03.999588 6019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:39111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:43757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:43757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:17:03.999861 6019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:17:04.000077 6019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:17:04.004944 6027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:04.004933 6024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:04.005188 6025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:04.005452 6019 server_base.cc:1034] running on GCE node
I20260620 23:17:04.005677 6019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:17:04.005952 6019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:17:04.007120 6019 hybrid_clock.cc:648] HybridClock initialized: now 1781997424007114 us; error 34 us; skew 500 ppm
Jun 20 23:17:04 dist-test-slave-pswb krb5kdc[6006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997424, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:17:04.010406 6019 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:17:04.010948 6019 webserver.cc:460] Webserver started at http://127.16.79.62:37349/ using document root <none> and password file <none>
I20260620 23:17:04.011221 6019 fs_manager.cc:359] Metadata directory not provided
I20260620 23:17:04.011274 6019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:17:04.011417 6019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:17:04.012645 6019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "519baf1ae05944b79621c661f5f2d919"
format_stamp: "Formatted at 2026-06-20 23:17:04 on dist-test-slave-pswb"
server_key: "3da70f3b5d5cc133c9cfaa454b5c596f"
server_key_iv: "3abda5f8bb4c11768f7d003165d753db"
server_key_version: "encryptionkey@0"
I20260620 23:17:04.013101 6019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "519baf1ae05944b79621c661f5f2d919"
format_stamp: "Formatted at 2026-06-20 23:17:04 on dist-test-slave-pswb"
server_key: "3da70f3b5d5cc133c9cfaa454b5c596f"
server_key_iv: "3abda5f8bb4c11768f7d003165d753db"
server_key_version: "encryptionkey@0"
I20260620 23:17:04.014674 6019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:17:04.015563 6034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:17:04.015893 6019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:17:04.015995 6019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "519baf1ae05944b79621c661f5f2d919"
format_stamp: "Formatted at 2026-06-20 23:17:04 on dist-test-slave-pswb"
server_key: "3da70f3b5d5cc133c9cfaa454b5c596f"
server_key_iv: "3abda5f8bb4c11768f7d003165d753db"
server_key_version: "encryptionkey@0"
I20260620 23:17:04.016083 6019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:17:04.019444 6019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:17:04.021701 6019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:17:04.021816 6019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:17:04.025889 6019 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:43757
I20260620 23:17:04.025949 6086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:43757 every 8 connection(s)
I20260620 23:17:04.026254 6019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:17:04.027124 6087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:17:04.029296 6087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919: Bootstrap starting.
I20260620 23:17:04.029855 6087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919: Neither blocks nor log segments found. Creating new log.
I20260620 23:17:04.030143 6087 log.cc:824] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919: Log is configured to *not* fsync() on all Append() calls
I20260620 23:17:04.030546 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 6019
I20260620 23:17:04.030685 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:17:04.030908 16700 external_mini_cluster.cc:1428] Setting key 178d25117776eb19e3e5806f61767345
I20260620 23:17:04.030972 6087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919: No bootstrap required, opened a new log
I20260620 23:17:04.033984 6087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519baf1ae05944b79621c661f5f2d919" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 43757 } }
I20260620 23:17:04.034325 6087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:17:04.034379 6087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519baf1ae05944b79621c661f5f2d919, State: Initialized, Role: FOLLOWER
I20260620 23:17:04.034641 6087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519baf1ae05944b79621c661f5f2d919" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 43757 } }
I20260620 23:17:04.034799 6087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:17:04.034844 6087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:17:04.034935 6087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:17:04.035421 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:04.031389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53145 (local address 127.16.79.62:43757)
0620 23:17:04.031598 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:04.031605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:04.031671 (+ 66us) server_negotiation.cc:407] Connection header received
0620 23:17:04.033798 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:04.033807 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:04.033840 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:04.033969 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:04.034932 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:04.034936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:04.034937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:17:04.034941 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:04.034984 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:17:04.034999 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:04.035067 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:04.035156 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 23:17:04.035195 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":101,"threads_started":1}
I20260620 23:17:04.036168 6087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519baf1ae05944b79621c661f5f2d919" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 43757 } }
I20260620 23:17:04.036300 6087 leader_election.cc:302] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 519baf1ae05944b79621c661f5f2d919; no voters:
I20260620 23:17:04.036466 6087 leader_election.cc:288] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:17:04.036460 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:17:04.036504 6092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:17:04.036643 6092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [term 1 LEADER]: Becoming Leader. State: Replica: 519baf1ae05944b79621c661f5f2d919, State: Running, Role: LEADER
I20260620 23:17:04.036707 6087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:17:04.036803 6092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519baf1ae05944b79621c661f5f2d919" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 43757 } }
I20260620 23:17:04.038854 6094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "519baf1ae05944b79621c661f5f2d919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519baf1ae05944b79621c661f5f2d919" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 43757 } } }
I20260620 23:17:04.038988 6094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:04.039440 6111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:17:04.039546 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:04.038385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40921 (local address 127.16.79.62:43757)
0620 23:17:04.038427 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:04.038429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:04.038568 (+ 139us) server_negotiation.cc:407] Connection header received
0620 23:17:04.038665 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:04.038665 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:04.038671 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:04.038715 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:04.039293 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:04.039295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:04.039295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:17:04.039299 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:04.039316 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:17:04.039321 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:04.039368 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:04.039464 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 23:17:04.039478 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:17:04.039695 6093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519baf1ae05944b79621c661f5f2d919. Latest consensus state: current_term: 1 leader_uuid: "519baf1ae05944b79621c661f5f2d919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519baf1ae05944b79621c661f5f2d919" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 43757 } } }
I20260620 23:17:04.039935 6093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919 [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:04.040444 6111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:17:04.042326 6111 catalog_manager.cc:1347] Generated new cluster ID: 874836dd70c84f2e85408cf7492ed51c
I20260620 23:17:04.042400 6111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:17:04.052366 6111 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:17:04.052532 6111 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:17:04.056172 6111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 519baf1ae05944b79621c661f5f2d919: Generated new TSK 0
I20260620 23:17:04.056326 6111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:17:04.061519 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 6019
2026-06-20T23:17:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:17:04 dist-test-slave-pswb krb5kdc[6118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:17:04 dist-test-slave-pswb krb5kdc[6118](info): set up 2 sockets
Jun 20 23:17:04 dist-test-slave-pswb krb5kdc[6118](info): commencing operation
krb5kdc: starting...
W20260620 23:17:05.116937 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:17:05 dist-test-slave-pswb krb5kdc[6118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:17:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:17:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:17:05.248373 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:41019
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:41019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:17:05.327958 6131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:17:05.328145 6131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:17:05.328166 6131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:17:05.329293 6131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:17:05.329334 6131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:17:05.329348 6131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:17:05.329360 6131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:17:05.329372 6131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:17:05.330796 6131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:41019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:41019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:17:05.331023 6131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:17:05.331244 6131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:17:05.336083 6136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:05.336243 6131 server_base.cc:1034] running on GCE node
W20260620 23:17:05.336072 6139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:05.336052 6137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:05.336571 6131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:17:05.336894 6131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:17:05.338083 6131 hybrid_clock.cc:648] HybridClock initialized: now 1781997425338084 us; error 57 us; skew 500 ppm
Jun 20 23:17:05 dist-test-slave-pswb krb5kdc[6118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997425, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:17:05.344985 6131 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:17:05.345448 6131 webserver.cc:460] Webserver started at http://127.16.79.62:35571/ using document root <none> and password file <none>
I20260620 23:17:05.345683 6131 fs_manager.cc:359] Metadata directory not provided
I20260620 23:17:05.345729 6131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:17:05.345853 6131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:17:05.346860 6131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "30517b9c419b4fa38935a15d835d47de"
format_stamp: "Formatted at 2026-06-20 23:17:05 on dist-test-slave-pswb"
server_key: "e66f07485dd95cfddafc4b13b68fe204"
server_key_iv: "15aecc2d2f19cb762b3ea81fa8d744b8"
server_key_version: "encryptionkey@0"
I20260620 23:17:05.347213 6131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "30517b9c419b4fa38935a15d835d47de"
format_stamp: "Formatted at 2026-06-20 23:17:05 on dist-test-slave-pswb"
server_key: "e66f07485dd95cfddafc4b13b68fe204"
server_key_iv: "15aecc2d2f19cb762b3ea81fa8d744b8"
server_key_version: "encryptionkey@0"
I20260620 23:17:05.348529 6131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:17:05.349406 6146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:17:05.349745 6131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:17:05.349915 6131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "30517b9c419b4fa38935a15d835d47de"
format_stamp: "Formatted at 2026-06-20 23:17:05 on dist-test-slave-pswb"
server_key: "e66f07485dd95cfddafc4b13b68fe204"
server_key_iv: "15aecc2d2f19cb762b3ea81fa8d744b8"
server_key_version: "encryptionkey@0"
I20260620 23:17:05.350021 6131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:17:05.358741 6131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:17:05.360888 6131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:17:05.361004 6131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:17:05.365092 6131 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:41019
I20260620 23:17:05.365123 6198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:41019 every 8 connection(s)
I20260620 23:17:05.365470 6131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:17:05.366272 6199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:17:05.368366 6199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de: Bootstrap starting.
I20260620 23:17:05.368959 6199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de: Neither blocks nor log segments found. Creating new log.
I20260620 23:17:05.369231 6199 log.cc:824] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de: Log is configured to *not* fsync() on all Append() calls
I20260620 23:17:05.370132 6199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de: No bootstrap required, opened a new log
I20260620 23:17:05.371682 6199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30517b9c419b4fa38935a15d835d47de" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41019 } }
I20260620 23:17:05.371834 6199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:17:05.371882 6199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30517b9c419b4fa38935a15d835d47de, State: Initialized, Role: FOLLOWER
I20260620 23:17:05.372040 6199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30517b9c419b4fa38935a15d835d47de" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41019 } }
I20260620 23:17:05.372134 6199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:17:05.372177 6199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:17:05.372225 6199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:17:05.373096 6199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30517b9c419b4fa38935a15d835d47de" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41019 } }
I20260620 23:17:05.373178 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 6131
I20260620 23:17:05.373210 6199 leader_election.cc:302] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30517b9c419b4fa38935a15d835d47de; no voters:
I20260620 23:17:05.373330 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:17:05.373404 6199 leader_election.cc:288] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:17:05.373461 6202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:17:05.373560 16700 external_mini_cluster.cc:1428] Setting key cc452d6277f376d7f0d661399ca5c82e
I20260620 23:17:05.373612 6202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [term 1 LEADER]: Becoming Leader. State: Replica: 30517b9c419b4fa38935a15d835d47de, State: Running, Role: LEADER
I20260620 23:17:05.373643 6199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:17:05.373690 6202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30517b9c419b4fa38935a15d835d47de" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41019 } }
I20260620 23:17:05.374014 6204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30517b9c419b4fa38935a15d835d47de. Latest consensus state: current_term: 1 leader_uuid: "30517b9c419b4fa38935a15d835d47de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30517b9c419b4fa38935a15d835d47de" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41019 } } }
I20260620 23:17:05.374116 6204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:05.374773 6206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:17:05.375115 6203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30517b9c419b4fa38935a15d835d47de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30517b9c419b4fa38935a15d835d47de" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 41019 } } }
I20260620 23:17:05.375381 6206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:17:05.375397 6203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:05.377584 6206 catalog_manager.cc:1347] Generated new cluster ID: 51c81d2f43494707aaf47f011e4f44a7
I20260620 23:17:05.377640 6206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:17:05 dist-test-slave-pswb krb5kdc[6118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:17:05.382133 6206 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:17:05.382866 6206 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:17:05.386250 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:05.374694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59769 (local address 127.16.79.62:41019)
0620 23:17:05.374918 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:05.374928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:05.374962 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:17:05.375335 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:05.375344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:05.375404 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:05.375516 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:05.376470 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:05.377797 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:05.378805 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:05.379412 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:05.379495 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:05.382469 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:05.382473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:05.382474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:05.382486 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:05.384468 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:05.384854 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:05.384857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:05.384857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:05.384959 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:05.385122 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:05.385123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:05.385124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:05.385357 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:05.385504 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:05.385888 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 23:17:05.386024 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
I20260620 23:17:05.388422 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:17:05.394757 6206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30517b9c419b4fa38935a15d835d47de: Generated new TSK 0
I20260620 23:17:05.394930 6206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:17:05.398062 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:05.389860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37653 (local address 127.16.79.62:41019)
0620 23:17:05.389916 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:05.389918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:05.389981 (+ 63us) server_negotiation.cc:407] Connection header received
0620 23:17:05.390060 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:05.390061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:05.390071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:05.390132 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:05.390890 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:05.391625 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:05.392866 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:05.393229 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:05.393331 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:05.395078 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:05.395079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:05.395080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:05.395087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:05.396623 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:05.397155 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:05.397156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:05.397157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:05.397243 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:05.397499 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:05.397500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:05.397501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:05.397615 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:05.397678 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:05.397889 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 23:17:05.397981 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:17:05.405632 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 6131
2026-06-20T23:17:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:17:05 dist-test-slave-pswb krb5kdc[6230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:17:05 dist-test-slave-pswb krb5kdc[6230](info): set up 2 sockets
Jun 20 23:17:05 dist-test-slave-pswb krb5kdc[6230](info): commencing operation
krb5kdc: starting...
W20260620 23:17:06.485301 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:17:06 dist-test-slave-pswb krb5kdc[6230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:17:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:17:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:17:06.612632 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36781
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:36781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:17:06.692904 6243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:17:06.693105 6243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:17:06.693128 6243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:17:06.694285 6243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:17:06.694331 6243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:17:06.694346 6243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:17:06.694360 6243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:17:06.694371 6243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:17:06.695840 6243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:36781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:36781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:17:06.696100 6243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:17:06.696328 6243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:17:06.701347 6251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:06.701390 6249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:06.701424 6248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:06.701905 6243 server_base.cc:1034] running on GCE node
I20260620 23:17:06.702418 6243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:17:06.702862 6243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:17:06.704088 6243 hybrid_clock.cc:648] HybridClock initialized: now 1781997426704073 us; error 29 us; skew 500 ppm
Jun 20 23:17:06 dist-test-slave-pswb krb5kdc[6230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997426, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:17:06.707942 6243 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:17:06.708437 6243 webserver.cc:460] Webserver started at http://127.16.79.62:41291/ using document root <none> and password file <none>
I20260620 23:17:06.708665 6243 fs_manager.cc:359] Metadata directory not provided
I20260620 23:17:06.708765 6243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:17:06.708916 6243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:17:06.709985 6243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "255fd03294df465380587f2b15c447d4"
format_stamp: "Formatted at 2026-06-20 23:17:06 on dist-test-slave-pswb"
server_key: "a32a1ef7d9086f1c24b93c03c4142f73"
server_key_iv: "2cd122dd990a783fcfc49a0f7ef3acae"
server_key_version: "encryptionkey@0"
I20260620 23:17:06.710343 6243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "255fd03294df465380587f2b15c447d4"
format_stamp: "Formatted at 2026-06-20 23:17:06 on dist-test-slave-pswb"
server_key: "a32a1ef7d9086f1c24b93c03c4142f73"
server_key_iv: "2cd122dd990a783fcfc49a0f7ef3acae"
server_key_version: "encryptionkey@0"
I20260620 23:17:06.711793 6243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:17:06.713285 6258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:17:06.713613 6243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:17:06.713754 6243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "255fd03294df465380587f2b15c447d4"
format_stamp: "Formatted at 2026-06-20 23:17:06 on dist-test-slave-pswb"
server_key: "a32a1ef7d9086f1c24b93c03c4142f73"
server_key_iv: "2cd122dd990a783fcfc49a0f7ef3acae"
server_key_version: "encryptionkey@0"
I20260620 23:17:06.713863 6243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:17:06.727137 6243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:17:06.729916 6243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:17:06.730094 6243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:17:06.734376 6243 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:36781
I20260620 23:17:06.734411 6310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:36781 every 8 connection(s)
I20260620 23:17:06.734745 6243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:17:06.735612 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:17:06.737375 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 6243
I20260620 23:17:06.737506 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:17:06.737697 16700 external_mini_cluster.cc:1428] Setting key 890034ddf32245360e931629ee3e0559
I20260620 23:17:06.738408 6311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4: Bootstrap starting.
I20260620 23:17:06.739013 6311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4: Neither blocks nor log segments found. Creating new log.
I20260620 23:17:06.739650 6311 log.cc:824] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:17:06.740415 6311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4: No bootstrap required, opened a new log
I20260620 23:17:06.742331 6311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255fd03294df465380587f2b15c447d4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36781 } }
I20260620 23:17:06.742488 6311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:17:06.742551 6311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 255fd03294df465380587f2b15c447d4, State: Initialized, Role: FOLLOWER
I20260620 23:17:06.742723 6311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255fd03294df465380587f2b15c447d4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36781 } }
I20260620 23:17:06.742818 6311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:17:06.742861 6311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:17:06.742933 6311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:17:06.743577 6311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255fd03294df465380587f2b15c447d4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36781 } }
I20260620 23:17:06.743701 6311 leader_election.cc:302] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 255fd03294df465380587f2b15c447d4; no voters:
I20260620 23:17:06.743904 6311 leader_election.cc:288] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:17:06.743925 6316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:17:06.744210 6311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:17:06.744091 6316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [term 1 LEADER]: Becoming Leader. State: Replica: 255fd03294df465380587f2b15c447d4, State: Running, Role: LEADER
I20260620 23:17:06.744397 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:06.738362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52339 (local address 127.16.79.62:36781)
0620 23:17:06.738593 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:06.738604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:06.738639 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:17:06.739406 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:06.739421 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:06.739488 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:06.739676 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:06.739947 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:06.741123 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:06.741985 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:06.742453 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:06.742611 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:06.743514 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:06.743519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:06.743519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:17:06.743747 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:06.743795 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:17:06.743814 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:06.743886 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:06.743972 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 23:17:06.744053 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260620 23:17:06.744393 6316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255fd03294df465380587f2b15c447d4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36781 } }
I20260620 23:17:06.745028 6317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "255fd03294df465380587f2b15c447d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255fd03294df465380587f2b15c447d4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36781 } } }
I20260620 23:17:06.745340 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:17:06.745460 6318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 255fd03294df465380587f2b15c447d4. Latest consensus state: current_term: 1 leader_uuid: "255fd03294df465380587f2b15c447d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "255fd03294df465380587f2b15c447d4" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 36781 } } }
I20260620 23:17:06.745572 6317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:06.745591 6318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4 [sys.catalog]: This master's current role is: LEADER
W20260620 23:17:06.746078 6331 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:17:06.746121 6331 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:17:06.746201 6332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:17:06.746893 6332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:17:06.748399 6332 catalog_manager.cc:1347] Generated new cluster ID: 6bb4d277e16e4510b5d3132238830f3b
I20260620 23:17:06.748447 6332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:17:06.750134 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:06.746777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54579 (local address 127.16.79.62:36781)
0620 23:17:06.746999 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:06.747001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:06.747009 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:17:06.747029 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:06.747030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:06.747040 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:06.747089 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:06.747259 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:06.747985 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:06.748873 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:06.749169 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:06.749245 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:06.749898 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:06.749899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:06.749900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:17:06.749904 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:06.749919 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:17:06.749924 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:06.749982 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:06.750043 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 23:17:06.750064 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260620 23:17:06.754833 6332 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:17:06.755656 6332 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:17:06.759037 6332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 255fd03294df465380587f2b15c447d4: Generated new TSK 0
I20260620 23:17:06.759315 6332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:17:06.764417 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 6243
2026-06-20T23:17:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:17:06 dist-test-slave-pswb krb5kdc[6342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:17:06 dist-test-slave-pswb krb5kdc[6342](info): set up 2 sockets
Jun 20 23:17:06 dist-test-slave-pswb krb5kdc[6342](info): commencing operation
krb5kdc: starting...
W20260620 23:17:07.825143 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:17:07 dist-test-slave-pswb krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:17:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:17:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:17:07.956301 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45505
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:45505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:17:08.040222 6355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:17:08.040473 6355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:17:08.040511 6355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:17:08.041644 6355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:17:08.041715 6355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:17:08.041742 6355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:17:08.041775 6355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:17:08.041797 6355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:17:08.043195 6355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:45505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:45505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:17:08.043463 6355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:17:08.043681 6355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:17:08.048730 6360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:08.048753 6363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:08.049134 6355 server_base.cc:1034] running on GCE node
W20260620 23:17:08.048730 6361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:08.049448 6355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:17:08.049770 6355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:17:08.050952 6355 hybrid_clock.cc:648] HybridClock initialized: now 1781997428050896 us; error 104 us; skew 500 ppm
Jun 20 23:17:08 dist-test-slave-pswb krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997428, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:17:08.054255 6355 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:17:08.054873 6355 webserver.cc:460] Webserver started at http://127.16.79.62:41997/ using document root <none> and password file <none>
I20260620 23:17:08.055109 6355 fs_manager.cc:359] Metadata directory not provided
I20260620 23:17:08.055179 6355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:17:08.055330 6355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:17:08.056372 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "009763c7eb564b61a04c5ee8478901fe"
format_stamp: "Formatted at 2026-06-20 23:17:08 on dist-test-slave-pswb"
server_key: "c3e223075931275ca34d5dc4a800ca29"
server_key_iv: "d78eaddab159a07898f76e3c50708eaa"
server_key_version: "encryptionkey@0"
I20260620 23:17:08.056783 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "009763c7eb564b61a04c5ee8478901fe"
format_stamp: "Formatted at 2026-06-20 23:17:08 on dist-test-slave-pswb"
server_key: "c3e223075931275ca34d5dc4a800ca29"
server_key_iv: "d78eaddab159a07898f76e3c50708eaa"
server_key_version: "encryptionkey@0"
I20260620 23:17:08.058091 6355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:17:08.058871 6370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:17:08.059068 6355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:17:08.059165 6355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "009763c7eb564b61a04c5ee8478901fe"
format_stamp: "Formatted at 2026-06-20 23:17:08 on dist-test-slave-pswb"
server_key: "c3e223075931275ca34d5dc4a800ca29"
server_key_iv: "d78eaddab159a07898f76e3c50708eaa"
server_key_version: "encryptionkey@0"
I20260620 23:17:08.059248 6355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:17:08.062493 6355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:17:08.064656 6355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:17:08.064823 6355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:17:08.069190 6355 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:45505
I20260620 23:17:08.069232 6422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:45505 every 8 connection(s)
I20260620 23:17:08.069535 6355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:17:08.070600 6423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:17:08.071123 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 6355
I20260620 23:17:08.071240 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:17:08.071431 16700 external_mini_cluster.cc:1428] Setting key e9c8092d731b0d76896777ee822ae003
I20260620 23:17:08.073519 6423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe: Bootstrap starting.
I20260620 23:17:08.073902 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:08.071981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45357 (local address 127.16.79.62:45505)
0620 23:17:08.072153 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:08.072161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:08.072186 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:17:08.072618 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:08.072625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:08.072655 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:08.072830 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:08.073453 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:08.073456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:08.073456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:17:08.073464 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:08.073507 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:17:08.073516 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:08.073575 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:08.073661 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 23:17:08.073722 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260620 23:17:08.074189 6423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe: Neither blocks nor log segments found. Creating new log.
I20260620 23:17:08.074496 6423 log.cc:824] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe: Log is configured to *not* fsync() on all Append() calls
I20260620 23:17:08.074972 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:17:08.075161 6423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe: No bootstrap required, opened a new log
I20260620 23:17:08.077214 6423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009763c7eb564b61a04c5ee8478901fe" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45505 } }
I20260620 23:17:08.077391 6423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:17:08.077469 6423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009763c7eb564b61a04c5ee8478901fe, State: Initialized, Role: FOLLOWER
I20260620 23:17:08.077546 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:08.076427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47383 (local address 127.16.79.62:45505)
0620 23:17:08.076462 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:08.076463 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:08.076584 (+ 121us) server_negotiation.cc:407] Connection header received
0620 23:17:08.076608 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:08.076609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:08.076613 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:08.076712 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:08.077305 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:08.077306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:08.077306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:17:08.077310 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:08.077323 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:17:08.077327 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:08.077395 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:08.077459 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 23:17:08.077471 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:17:08.077605 6423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009763c7eb564b61a04c5ee8478901fe" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45505 } }
I20260620 23:17:08.077694 6423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:17:08.077733 6423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:17:08.077790 6423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:17:08.078271 6423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009763c7eb564b61a04c5ee8478901fe" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45505 } }
I20260620 23:17:08.078387 6423 leader_election.cc:302] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 009763c7eb564b61a04c5ee8478901fe; no voters:
I20260620 23:17:08.078547 6423 leader_election.cc:288] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:17:08.078589 6433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:17:08.078778 6423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:17:08.078775 6433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [term 1 LEADER]: Becoming Leader. State: Replica: 009763c7eb564b61a04c5ee8478901fe, State: Running, Role: LEADER
I20260620 23:17:08.078913 6433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009763c7eb564b61a04c5ee8478901fe" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45505 } }
I20260620 23:17:08.079418 6435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 009763c7eb564b61a04c5ee8478901fe. Latest consensus state: current_term: 1 leader_uuid: "009763c7eb564b61a04c5ee8478901fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009763c7eb564b61a04c5ee8478901fe" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45505 } } }
I20260620 23:17:08.079519 6435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:08.079730 6434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "009763c7eb564b61a04c5ee8478901fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009763c7eb564b61a04c5ee8478901fe" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 45505 } } }
I20260620 23:17:08.079758 6442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:17:08.080006 6434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:08.080368 6442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:17:08.081846 6442 catalog_manager.cc:1347] Generated new cluster ID: d0343c32e9dd499cb6a5c0c30827eeb6
I20260620 23:17:08.081904 6442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:17:08.099125 6442 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:17:08.099328 6442 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:17:08.102495 6442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 009763c7eb564b61a04c5ee8478901fe: Generated new TSK 0
I20260620 23:17:08.102674 6442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:17:08.107929 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 6355
2026-06-20T23:17:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1342 ms)
[----------] 18 tests from AuthTokenIssuingTest (24158 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:17:08 dist-test-slave-pswb krb5kdc[6454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:17:08 dist-test-slave-pswb krb5kdc[6454](info): set up 2 sockets
Jun 20 23:17:08 dist-test-slave-pswb krb5kdc[6454](info): commencing operation
krb5kdc: starting...
W20260620 23:17:09.152946 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:17:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:17:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:17:09.271476 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:40419
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:40419,127.16.79.61:39885,127.16.79.60:34721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:17:09.351218 6467 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:17:09.351444 6467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:17:09.351473 6467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:17:09.351516 6467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:17:09.352767 6467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:17:09.352857 6467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:17:09.352892 6467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:17:09.352926 6467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:17:09.352958 6467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:17:09.354480 6467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:40419,127.16.79.61:39885,127.16.79.60:34721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:40419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:17:09.354748 6467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:17:09.354977 6467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:17:09.359737 6475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:09.360008 6472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:09.360103 6473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:09.360045 6467 server_base.cc:1034] running on GCE node
I20260620 23:17:09.360450 6467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:17:09.360774 6467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:17:09.361954 6467 hybrid_clock.cc:648] HybridClock initialized: now 1781997429361887 us; error 91 us; skew 500 ppm
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:17:09.364372 6467 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:17:09.364902 6467 webserver.cc:460] Webserver started at http://127.16.79.62:40009/ using document root <none> and password file <none>
I20260620 23:17:09.365139 6467 fs_manager.cc:359] Metadata directory not provided
I20260620 23:17:09.365185 6467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:17:09.365311 6467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:17:09.366353 6467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "d8cd64cba42949e3ad600fe220d17d8e"
format_stamp: "Formatted at 2026-06-20 23:17:09 on dist-test-slave-pswb"
server_key: "7ca3cff166bbff5f6ff71fbad59782ec"
server_key_iv: "f4982d83a8829bbb77aebb576f31df7f"
server_key_version: "encryptionkey@0"
I20260620 23:17:09.366704 6467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "d8cd64cba42949e3ad600fe220d17d8e"
format_stamp: "Formatted at 2026-06-20 23:17:09 on dist-test-slave-pswb"
server_key: "7ca3cff166bbff5f6ff71fbad59782ec"
server_key_iv: "f4982d83a8829bbb77aebb576f31df7f"
server_key_version: "encryptionkey@0"
I20260620 23:17:09.368165 6467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:17:09.369354 6482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:17:09.369570 6467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:17:09.369656 6467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "d8cd64cba42949e3ad600fe220d17d8e"
format_stamp: "Formatted at 2026-06-20 23:17:09 on dist-test-slave-pswb"
server_key: "7ca3cff166bbff5f6ff71fbad59782ec"
server_key_iv: "f4982d83a8829bbb77aebb576f31df7f"
server_key_version: "encryptionkey@0"
I20260620 23:17:09.369714 6467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:17:09.385610 6467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:17:09.387735 6467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:17:09.387842 6467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:17:09.392228 6467 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:40419
I20260620 23:17:09.392308 6534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:40419 every 8 connection(s)
I20260620 23:17:09.392606 6467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:17:09.393596 6535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:17:09.395064 6535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:17:09.396346 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 6467
I20260620 23:17:09.396569 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:17:09.396844 16700 external_mini_cluster.cc:1428] Setting key 5689e5db4c91d57545dd3590ffbda8c6
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.79.61@KRBTEST.COM; defaulting to no policy
I20260620 23:17:09.409945 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.395684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40419 (local address 127.0.0.1:60390)
0620 23:17:09.395970 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.396039 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:09.397025 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:09.397342 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:09.397344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:09.397739 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:09.397976 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.397982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.399623 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.399624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.401200 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.401206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.402119 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.402121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.402256 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.402957 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:09.402968 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:09.404837 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:09.408178 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.408184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.408188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.408482 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.408833 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.408834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.408834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.408933 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.409401 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:09.409410 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:09.409499 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 23:17:09.409587 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:17:09.409732 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260620 23:17:09.409948 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.395844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.16.79.62:40419)
0620 23:17:09.395995 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.396007 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.396206 (+ 199us) server_negotiation.cc:407] Connection header received
0620 23:17:09.397215 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.397217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.397237 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.397336 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.398088 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.399517 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.401375 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.401960 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.402100 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.404976 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.404989 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.404990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.405020 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.408062 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.408577 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.408581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.408581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.408748 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.408984 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.408985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.408985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.409285 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.409370 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.409709 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.409802 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.61@KRBTEST.COM" created.
I20260620 23:17:09.410916 6535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:17:09.411396 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.411191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39885 (local address 127.0.0.1:54260)
0620 23:17:09.411269 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.411342 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 23:17:09.411607 6486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.79.61:39885: connect: Connection refused (error 111)
I20260620 23:17:09.412217 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.412098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39885 (local address 127.0.0.1:54270)
0620 23:17:09.412159 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.412189 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 23:17:09.412340 6535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.61:39885: Network error: Client connection negotiation failed: client connection to 127.16.79.61:39885: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.61.keytab.
Entry for principal kudu/127.16.79.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.61.keytab.
I20260620 23:17:09.427913 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.61:39885
--webserver_interface=127.16.79.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.61.keytab
--principal=kudu/127.16.79.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:40419,127.16.79.61:39885,127.16.79.60:34721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:17:09.461624 6535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } attempt: 1
I20260620 23:17:09.462148 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.462001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39885 (local address 127.0.0.1:54274)
0620 23:17:09.462071 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.462115 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 23:17:09.462633 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.462545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39885 (local address 127.0.0.1:54278)
0620 23:17:09.462587 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.462613 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 23:17:09.462806 6535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.61:39885: Network error: Client connection negotiation failed: client connection to 127.16.79.61:39885: connect: Connection refused (error 111)
W20260620 23:17:09.507546 6542 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:17:09.507809 6542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:17:09.507843 6542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:17:09.507875 6542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:17:09.509037 6542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:17:09.509102 6542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:17:09.509130 6542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:17:09.509161 6542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:17:09.509186 6542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:17:09.510639 6542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:40419,127.16.79.61:39885,127.16.79.60:34721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.61.keytab
--principal=kudu/127.16.79.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.61:39885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.79.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:17:09.510911 6542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:17:09.511133 6542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:17:09.515923 6548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:09.516178 6542 server_base.cc:1034] running on GCE node
W20260620 23:17:09.515987 6550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:09.515931 6547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:09.516571 6542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:17:09.516906 6542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:17:09.518062 6542 hybrid_clock.cc:648] HybridClock initialized: now 1781997429518045 us; error 38 us; skew 500 ppm
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:17:09.520639 6542 init.cc:377] Logged in from keytab as kudu/127.16.79.61@KRBTEST.COM (short username kudu)
I20260620 23:17:09.521132 6542 webserver.cc:460] Webserver started at http://127.16.79.61:34683/ using document root <none> and password file <none>
I20260620 23:17:09.521354 6542 fs_manager.cc:359] Metadata directory not provided
I20260620 23:17:09.521428 6542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:17:09.521579 6542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:17:09.522619 6542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/data/instance:
uuid: "b63bdb131a8041a5af926d73414baae1"
format_stamp: "Formatted at 2026-06-20 23:17:09 on dist-test-slave-pswb"
server_key: "374adb6358f5b8093b992ca9806462d5"
server_key_iv: "9ed710be7b5ab1eb255d00b8df76c309"
server_key_version: "encryptionkey@0"
I20260620 23:17:09.522976 6542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/wal/instance:
uuid: "b63bdb131a8041a5af926d73414baae1"
format_stamp: "Formatted at 2026-06-20 23:17:09 on dist-test-slave-pswb"
server_key: "374adb6358f5b8093b992ca9806462d5"
server_key_iv: "9ed710be7b5ab1eb255d00b8df76c309"
server_key_version: "encryptionkey@0"
I20260620 23:17:09.524674 6542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 23:17:09.525602 6557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:17:09.525923 6542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:17:09.526029 6542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/wal
uuid: "b63bdb131a8041a5af926d73414baae1"
format_stamp: "Formatted at 2026-06-20 23:17:09 on dist-test-slave-pswb"
server_key: "374adb6358f5b8093b992ca9806462d5"
server_key_iv: "9ed710be7b5ab1eb255d00b8df76c309"
server_key_version: "encryptionkey@0"
I20260620 23:17:09.526114 6542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:17:09.530987 6535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } attempt: 2
I20260620 23:17:09.531368 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.531235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39885 (local address 127.0.0.1:54286)
0620 23:17:09.531296 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.531336 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 23:17:09.531708 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.531628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39885 (local address 127.0.0.1:54288)
0620 23:17:09.531670 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.531692 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 23:17:09.531829 6535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.61:39885: Network error: Client connection negotiation failed: client connection to 127.16.79.61:39885: connect: Connection refused (error 111)
I20260620 23:17:09.559887 6542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:17:09.562764 6542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:17:09.562985 6542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:17:09.568580 6542 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.61:39885
I20260620 23:17:09.568753 6609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.61:39885 every 8 connection(s)
I20260620 23:17:09.569067 6542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
I20260620 23:17:09.570389 6610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:17:09.572590 6610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:17:09.573371 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 6542
I20260620 23:17:09.573488 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-1/wal/instance
I20260620 23:17:09.573662 16700 external_mini_cluster.cc:1428] Setting key 1d60f14972df922311b30683aa4e48ff
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.61@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.79.60@KRBTEST.COM; defaulting to no policy
I20260620 23:17:09.583560 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.573489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.16.79.62:40419)
0620 23:17:09.573541 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.573543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.573823 (+ 280us) server_negotiation.cc:407] Connection header received
0620 23:17:09.574423 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.574424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.574436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.574499 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.575127 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.575621 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.577196 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.577548 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.577621 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.580131 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.580133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.580133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.580141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.581821 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.582341 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.582343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.582343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.582456 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.582819 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.582820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.582820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.582955 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.583070 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.583413 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.583489 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:17:09.583920 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.573488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40419 (local address 127.0.0.1:60406)
0620 23:17:09.573684 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.573726 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:09.574301 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:09.574565 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:09.574569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:09.574826 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:09.575022 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.575026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.575713 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.575714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.577040 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.577043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.577666 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.577667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.577825 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.578431 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:09.578439 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:09.580029 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:09.581927 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.581933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.581936 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.582223 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.582609 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.582610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.582611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.582716 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.583049 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:09.583062 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:09.583255 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 23:17:09.583369 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:17:09.583596 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":123,"threads_started":1}
I20260620 23:17:09.584832 6610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.60@KRBTEST.COM" created.
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.61@KRBTEST.COM for kudu/127.16.79.61@KRBTEST.COM
I20260620 23:17:09.593518 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.585203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39885 (local address 127.0.0.1:54294)
0620 23:17:09.585262 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.585273 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:09.585317 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:09.585713 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:09.585714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:09.585893 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:09.585965 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.585967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.587108 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.587109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.587642 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.587643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.588310 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.588310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.588377 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.588837 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:09.588842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:09.590281 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:09.592314 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.592315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.592316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.592592 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.592854 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.592855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.592855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.592945 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.593280 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:09.593281 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:09.593317 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:17:09.593374 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 23:17:09.593443 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 23:17:09.593668 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.585244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54294 (local address 127.16.79.61:39885)
0620 23:17:09.585452 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.585458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.585492 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:17:09.585521 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.585523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.585542 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.585693 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.586127 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.587014 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.587727 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.588231 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.588307 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.590375 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.590384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.590384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.590392 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.592213 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.592676 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.592679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.592679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.592768 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.593006 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.593033 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.593034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.593222 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.593302 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.593431 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.593501 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":110,"threads_started":1}
I20260620 23:17:09.594389 6610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:17:09.594725 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.594622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39440)
0620 23:17:09.594669 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.594709 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 23:17:09.594856 6560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
I20260620 23:17:09.595223 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.595141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39442)
0620 23:17:09.595184 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.595207 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 23:17:09.595551 6610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:34721: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.60.keytab.
Entry for principal kudu/127.16.79.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.60.keytab.
I20260620 23:17:09.604430 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.60:34721
--webserver_interface=127.16.79.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.60.keytab
--principal=kudu/127.16.79.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:40419,127.16.79.61:39885,127.16.79.60:34721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:17:09.623020 6535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } attempt: 3
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for kudu/127.16.79.61@KRBTEST.COM
I20260620 23:17:09.630054 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.623399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.16.79.61:39885)
0620 23:17:09.623464 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.623466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.623477 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:17:09.623549 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.623550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.623561 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.623629 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.623922 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.624575 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.625403 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.625639 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.625726 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.627544 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.627545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.627546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.627552 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.628958 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.629276 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.629277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.629277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.629331 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.629555 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.629556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.629556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.629658 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.629732 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.629891 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.629956 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:17:09.630054 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.623328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39885 (local address 127.0.0.1:54306)
0620 23:17:09.623399 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.623433 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:09.623484 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:09.623633 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:09.623634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:09.623786 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:09.623845 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.623847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.624655 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.624656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.625298 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.625300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.625725 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.625726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.625807 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.626267 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:09.626273 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:09.627460 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:09.629037 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.629038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.629039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.629205 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.629409 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.629410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.629410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.629498 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.629726 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:09.629726 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:09.629771 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 23:17:09.629858 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:17:09.629956 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 23:17:09.630402 6535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:17:09.630728 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.630635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39446)
0620 23:17:09.630682 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.630708 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 23:17:09.631088 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.631000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39450)
0620 23:17:09.631046 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.631069 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 23:17:09.631203 6535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:34721: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
I20260620 23:17:09.644810 6610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } attempt: 1
I20260620 23:17:09.645350 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.645099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39462)
0620 23:17:09.645234 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.645304 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 23:17:09.645946 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.645803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39478)
0620 23:17:09.645880 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.645917 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:17:09.646080 6610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:34721: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
I20260620 23:17:09.662377 6535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } attempt: 1
I20260620 23:17:09.662847 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.662727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39480)
0620 23:17:09.662789 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.662824 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 23:17:09.663172 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.663084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39494)
0620 23:17:09.663127 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.663149 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 23:17:09.663254 6535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:34721: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
W20260620 23:17:09.685412 6617 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:17:09.685595 6617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:17:09.685611 6617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:17:09.685631 6617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:17:09.686748 6617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:17:09.686784 6617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:17:09.686798 6617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:17:09.686811 6617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:17:09.686847 6617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:17:09.688261 6617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:40419,127.16.79.61:39885,127.16.79.60:34721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.60.keytab
--principal=kudu/127.16.79.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.60:34721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.79.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:17:09.688481 6617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:17:09.688743 6617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:17:09.694013 6623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:09.694094 6617 server_base.cc:1034] running on GCE node
W20260620 23:17:09.694216 6622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:09.694366 6625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:09.694690 6617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:17:09.695029 6617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:17:09.696197 6617 hybrid_clock.cc:648] HybridClock initialized: now 1781997429696176 us; error 36 us; skew 500 ppm
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:17:09.699414 6617 init.cc:377] Logged in from keytab as kudu/127.16.79.60@KRBTEST.COM (short username kudu)
I20260620 23:17:09.699842 6617 webserver.cc:460] Webserver started at http://127.16.79.60:46149/ using document root <none> and password file <none>
I20260620 23:17:09.700084 6617 fs_manager.cc:359] Metadata directory not provided
I20260620 23:17:09.700131 6617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:17:09.700263 6617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:17:09.701324 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/data/instance:
uuid: "a059b8d0f84b4c0eb06da9639477a46e"
format_stamp: "Formatted at 2026-06-20 23:17:09 on dist-test-slave-pswb"
server_key: "27e4fe3d2fb8bbe26cfe944a755880a3"
server_key_iv: "007c6a361cba7318c25b051107d9c799"
server_key_version: "encryptionkey@0"
I20260620 23:17:09.701664 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/wal/instance:
uuid: "a059b8d0f84b4c0eb06da9639477a46e"
format_stamp: "Formatted at 2026-06-20 23:17:09 on dist-test-slave-pswb"
server_key: "27e4fe3d2fb8bbe26cfe944a755880a3"
server_key_iv: "007c6a361cba7318c25b051107d9c799"
server_key_version: "encryptionkey@0"
I20260620 23:17:09.703056 6617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:17:09.704010 6632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:17:09.704236 6617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:17:09.704341 6617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/wal
uuid: "a059b8d0f84b4c0eb06da9639477a46e"
format_stamp: "Formatted at 2026-06-20 23:17:09 on dist-test-slave-pswb"
server_key: "27e4fe3d2fb8bbe26cfe944a755880a3"
server_key_iv: "007c6a361cba7318c25b051107d9c799"
server_key_version: "encryptionkey@0"
I20260620 23:17:09.704418 6617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:17:09.714241 6610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } attempt: 2
I20260620 23:17:09.714643 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.714525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39506)
0620 23:17:09.714583 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.714618 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 23:17:09.714967 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.714877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39518)
0620 23:17:09.714922 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.714946 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 23:17:09.715131 6610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:34721: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34721: connect: Connection refused (error 111)
I20260620 23:17:09.720427 6617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:17:09.722697 6617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:17:09.722844 6617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:17:09.726981 6617 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.60:34721
I20260620 23:17:09.727010 6684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.60:34721 every 8 connection(s)
I20260620 23:17:09.727348 6617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
I20260620 23:17:09.728488 6685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:17:09.729846 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 6617
I20260620 23:17:09.729951 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997163808439-16700-0/minicluster-data/master-2/wal/instance
I20260620 23:17:09.730119 16700 external_mini_cluster.cc:1428] Setting key 0dced417059291c846d4be605f72aa89
I20260620 23:17:09.730237 6685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.60@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.60@KRBTEST.COM
I20260620 23:17:09.738390 6535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } attempt: 2
I20260620 23:17:09.739300 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.730914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.16.79.62:40419)
0620 23:17:09.730976 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.730978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.731162 (+ 184us) server_negotiation.cc:407] Connection header received
0620 23:17:09.731334 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.731335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.731347 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.731413 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.731931 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.732424 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.733341 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.733547 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.733632 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.735936 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.735937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.735938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.735952 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.737609 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.738063 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.738064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.738064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.738124 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.738403 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.738404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.738404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.738553 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.738619 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.739105 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.739183 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:17:09.739705 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.730856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:40419 (local address 127.0.0.1:60412)
0620 23:17:09.731071 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.731096 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:09.731251 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:09.731437 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:09.731440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:09.731686 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:09.731834 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.731837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.732507 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.732508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.733257 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.733259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.733663 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.733664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.733777 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.734233 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:09.734243 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:09.735809 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:09.737721 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.737727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.737728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.737989 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.738207 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.738207 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.738208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.738302 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.738782 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:09.738788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:09.738971 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 23:17:09.739083 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 23:17:09.739237 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260620 23:17:09.739714 6688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.730604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39526 (local address 127.16.79.60:34721)
0620 23:17:09.730772 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.730778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.730800 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:17:09.731249 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.731256 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.731274 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.731382 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.731995 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.732815 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.733541 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.733834 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.733927 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.736394 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.736402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.736402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.736408 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.738228 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.738603 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.738606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.738607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.738798 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.739048 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.739049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.739049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.739192 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.739309 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.739548 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.739618 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260620 23:17:09.740335 6685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for kudu/127.16.79.60@KRBTEST.COM
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.60@KRBTEST.COM for kudu/127.16.79.61@KRBTEST.COM
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:17:09.748989 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.738660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.16.79.60:34721)
0620 23:17:09.738791 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.738793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.738866 (+ 73us) server_negotiation.cc:407] Connection header received
0620 23:17:09.738926 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.738927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.738936 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.738993 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.739280 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.739958 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.741050 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.741351 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.741423 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.744581 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.744582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.744582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.744591 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.747116 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.748035 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.748036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.748036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.748103 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.748315 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.748315 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.748316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.748460 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.748540 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.748748 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.748833 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260620 23:17:09.748965 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.738613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39540)
0620 23:17:09.738806 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.738818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:09.738870 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:09.738994 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:09.738994 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:09.739166 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:09.739200 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.739201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.740204 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.740205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.740972 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.740974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.741429 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.741430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.741483 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.742565 (+ 1082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:09.742570 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:09.744462 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:09.747759 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.747760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.747761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.747950 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.748172 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.748172 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.748173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.748237 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.748533 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:09.748534 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:09.748587 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 23:17:09.748670 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:17:09.748847 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 23:17:09.749001 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.740641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.16.79.61:39885)
0620 23:17:09.740686 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.740708 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.740725 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:17:09.740788 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.740790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.740801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.740857 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.741099 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.741642 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.742598 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.743082 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.743184 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.746051 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.746053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.746054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.746065 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.747667 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.748085 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.748086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.748087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.748142 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.748360 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.748361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.748369 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.748514 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.748584 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.748815 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.748906 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:17:09.749066 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.740643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.16.79.62:40419)
0620 23:17:09.741074 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.741076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.741081 (+ 5us) server_negotiation.cc:407] Connection header received
0620 23:17:09.741096 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.741096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.741104 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.741143 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.741766 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.742445 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.743126 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.743328 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.743417 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.745997 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.745998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.745999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.746010 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.747532 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.748013 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.748014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.748015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.748070 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.748276 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.748277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.748277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.748390 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.748519 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.748770 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.748922 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":416}
I20260620 23:17:09.749305 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.740583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:39885 (local address 127.0.0.1:54310)
0620 23:17:09.740642 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.740652 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:09.740689 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:09.740861 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:09.740861 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:09.740987 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:09.741021 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.741022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.741735 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.741736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.742508 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.742511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.743326 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.743328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.743420 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.743990 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:09.743996 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:09.745896 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:09.747734 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.747735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.747736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.747972 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.748224 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.748225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.748225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.748294 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.748618 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:09.748619 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:09.748668 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:17:09.749082 (+ 414us) client_negotiation.cc:240] Negotiation successful
0620 23:17:09.749194 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:17:09.750300 6685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:17:09.751623 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e: Bootstrap starting.
I20260620 23:17:09.752339 6535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e: Neither blocks nor log segments found. Creating new log.
I20260620 23:17:09.752664 6535 log.cc:824] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:17:09.753652 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e: No bootstrap required, opened a new log
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.61@KRBTEST.COM
I20260620 23:17:09.755842 6535 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } }
I20260620 23:17:09.755991 6535 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8cd64cba42949e3ad600fe220d17d8e, State: Initialized, Role: FOLLOWER
I20260620 23:17:09.756069 6535 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } }
I20260620 23:17:09.756593 6692 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } } }
I20260620 23:17:09.756773 6692 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:09.756954 6535 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.60@KRBTEST.COM for kudu/127.16.79.60@KRBTEST.COM
I20260620 23:17:09.758311 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.750196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54318 (local address 127.16.79.61:39885)
0620 23:17:09.750235 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.750238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.750285 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:17:09.750358 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.750358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.750369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.750425 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.751061 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.751541 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.752246 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.752547 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.752633 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.755082 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.755084 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.755084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.755091 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.756804 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.757257 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.757258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.757259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.757320 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.757549 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.757549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.757550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.757703 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.757839 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.758134 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.758212 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:17:09.759714 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 23:17:09.759716 6703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:17:09.759783 6703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:17:09.760731 6509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d8cd64cba42949e3ad600fe220d17d8e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60428
I20260620 23:17:09.760843 6509 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:17:09.760888 6509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:17:09.761767 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.750506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39542)
0620 23:17:09.750608 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.750617 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:09.750651 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:09.750779 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:09.750780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:09.750953 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:09.750995 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.750997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.751846 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.751847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.752578 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.752580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.753114 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.753115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.753164 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.756890 (+ 3726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:09.756896 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:09.758737 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:09.760628 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.760628 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.760629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.760846 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.761026 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.761027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.761027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.761076 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.761490 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:09.761491 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:09.761531 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:17:09.761590 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 23:17:09.761685 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 23:17:09.761946 6509 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } }
I20260620 23:17:09.762081 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.750568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.16.79.60:34721)
0620 23:17:09.750643 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.750645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.750656 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:17:09.750704 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.750705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.750717 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.750773 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.751104 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.751763 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.752669 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.753020 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.753113 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.758861 (+ 5748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.758862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.758862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.758869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.760547 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.760922 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.760923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.760923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.760968 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.761130 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.761130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.761131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.761386 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.761466 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.761940 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.762006 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:17:09.762172 6509 leader_election.cc:288] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [CANDIDATE]: Term 1 election: Requested vote from peers b63bdb131a8041a5af926d73414baae1 (127.16.79.61:39885), a059b8d0f84b4c0eb06da9639477a46e (127.16.79.60:34721)
I20260620 23:17:09.760243 6467 master_runner.cc:186] Error getting master registration for 127.16.79.61:39885: OK, instance_id { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" instance_seqno: 1781997429567008 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:17:09.762938 6467 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:17:09.763384 6685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e: Bootstrap starting.
I20260620 23:17:09.763942 6685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e: Neither blocks nor log segments found. Creating new log.
I20260620 23:17:09.764220 6685 log.cc:824] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:17:09.765026 6685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e: No bootstrap required, opened a new log
I20260620 23:17:09.766947 6685 raft_consensus.cc:348] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } }
I20260620 23:17:09.767082 6685 raft_consensus.cc:729] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a059b8d0f84b4c0eb06da9639477a46e, State: Initialized, Role: FOLLOWER
I20260620 23:17:09.767161 6685 consensus_queue.cc:260] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } }
I20260620 23:17:09.767391 6710 sys_catalog.cc:455] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } } }
I20260620 23:17:09.767439 6685 sys_catalog.cc:564] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:17:09.767485 6710 sys_catalog.cc:458] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:09.767591 6659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8cd64cba42949e3ad600fe220d17d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a059b8d0f84b4c0eb06da9639477a46e"
I20260620 23:17:09.767694 6659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:17:09.768851 6659 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8cd64cba42949e3ad600fe220d17d8e in term 1.
I20260620 23:17:09.769203 6485 leader_election.cc:302] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a059b8d0f84b4c0eb06da9639477a46e, d8cd64cba42949e3ad600fe220d17d8e; no voters:
I20260620 23:17:09.769336 6692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:17:09.769505 6692 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [term 1 LEADER]: Becoming Leader. State: Replica: d8cd64cba42949e3ad600fe220d17d8e, State: Running, Role: LEADER
I20260620 23:17:09.769595 6692 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } }
I20260620 23:17:09.769989 6722 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8cd64cba42949e3ad600fe220d17d8e. Latest consensus state: current_term: 1 leader_uuid: "d8cd64cba42949e3ad600fe220d17d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } } }
I20260620 23:17:09.770165 6722 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [sys.catalog]: This master's current role is: LEADER
W20260620 23:17:09.770155 6721 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:17:09.770203 6721 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:17:09.770485 6725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:17:09.770933 6725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:17:09.772522 6617 master_runner.cc:186] Error getting master registration for 127.16.79.61:39885: OK, instance_id { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" instance_seqno: 1781997429567008 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:17:09.775709 6617 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:17:09.779754 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.770745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.16.79.60:34721)
0620 23:17:09.770777 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.770778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.772023 (+ 1245us) server_negotiation.cc:407] Connection header received
0620 23:17:09.772117 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.772118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.772127 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.772155 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.773024 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.773754 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.774953 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.775237 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.775279 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.776945 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.776946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.776947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.776953 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.778581 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.778959 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.778960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.778961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.779010 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.779198 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.779199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.779200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.779311 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.779398 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.779598 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.779669 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 23:17:09.785887 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.771036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.16.79.62:40419)
0620 23:17:09.771080 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.771082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.776125 (+ 5043us) server_negotiation.cc:407] Connection header received
0620 23:17:09.776269 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.776270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.776305 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.776336 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.777306 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.778003 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.779605 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.779900 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.779942 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.782256 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.782258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.782258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.782265 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.784025 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.784865 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.784866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.784866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.784924 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.785203 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.785204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.785204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.785344 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.785435 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.785723 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.785797 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:17:09.788539 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.771204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.16.79.61:39885)
0620 23:17:09.771242 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.771244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.780233 (+ 8989us) server_negotiation.cc:407] Connection header received
0620 23:17:09.780334 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.780336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.780347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.780377 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.781050 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.781885 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.783084 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.783413 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.783456 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.785297 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.785299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.785299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.785307 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.787204 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.787622 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.787624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.787624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.787699 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.787923 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.787923 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.787924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.788050 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.788148 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.788350 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.788448 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:17:09.789750 6659 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [term 1 FOLLOWER]: Refusing update from remote peer d8cd64cba42949e3ad600fe220d17d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:17:09.790156 6722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:17:09.791519 6710 sys_catalog.cc:455] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8cd64cba42949e3ad600fe220d17d8e. Latest consensus state: current_term: 1 leader_uuid: "d8cd64cba42949e3ad600fe220d17d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } } }
I20260620 23:17:09.791647 6710 sys_catalog.cc:458] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:09.797863 6692 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8cd64cba42949e3ad600fe220d17d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } } }
I20260620 23:17:09.797989 6692 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:09.798591 6725 catalog_manager.cc:1347] Generated new cluster ID: 3770495f11fd489da1c930c27482b285
I20260620 23:17:09.798506 6710 sys_catalog.cc:455] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8cd64cba42949e3ad600fe220d17d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } } }
I20260620 23:17:09.798645 6725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:17:09.798669 6710 sys_catalog.cc:458] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:09.799042 6730 mvcc.cc:205] Tried to move back new op lower bound from 7299061472415420416 to 7299061472336760832. Current Snapshot: MvccSnapshot[applied={T|T < 7299061472415420416}]
I20260620 23:17:09.806296 6610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } attempt: 3
Jun 20 23:17:09 dist-test-slave-pswb krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997429, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.61@KRBTEST.COM for kudu/127.16.79.60@KRBTEST.COM
I20260620 23:17:09.814917 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.806542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34721 (local address 127.0.0.1:39558)
0620 23:17:09.806619 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:09.806630 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:09.806667 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:09.806915 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:09.806915 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:09.807092 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:09.807156 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.807158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.808059 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.808060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.808892 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:09.808895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.809376 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.809377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:09.809429 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.809929 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:09.809935 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:09.811422 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:09.813890 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.813891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.813892 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.814103 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.814297 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:09.814298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:09.814298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:09.814351 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:09.814612 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:09.814613 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:09.814654 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:17:09.814714 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 23:17:09.814805 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:17:09.815244 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:09.806743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.16.79.60:34721)
0620 23:17:09.806778 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.806779 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.806786 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:17:09.806814 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.806814 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.806823 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.806868 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.807279 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.807963 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.808982 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:09.809270 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:09.809327 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:09.811757 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:09.811759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:09.811760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:09.811771 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:09.813779 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.814180 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.814181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.814181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.814234 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:09.814413 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:09.814414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:09.814414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:09.814536 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:09.814589 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:09.815088 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 23:17:09.815157 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:17:09.816768 6610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1: Bootstrap starting.
I20260620 23:17:09.817536 6610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1: Neither blocks nor log segments found. Creating new log.
I20260620 23:17:09.817871 6610 log.cc:824] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:17:09.818720 6610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1: No bootstrap required, opened a new log
I20260620 23:17:09.820907 6610 raft_consensus.cc:348] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } }
I20260620 23:17:09.821041 6610 raft_consensus.cc:729] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63bdb131a8041a5af926d73414baae1, State: Initialized, Role: FOLLOWER
I20260620 23:17:09.821115 6610 consensus_queue.cc:260] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } }
I20260620 23:17:09.821435 6610 sys_catalog.cc:564] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:17:09.821516 6584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:17:09.822531 6584 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [term 1 FOLLOWER]: Refusing update from remote peer d8cd64cba42949e3ad600fe220d17d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:17:09.822837 6722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:17:09.823226 6583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8cd64cba42949e3ad600fe220d17d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b63bdb131a8041a5af926d73414baae1"
I20260620 23:17:09.824004 6583 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8cd64cba42949e3ad600fe220d17d8e in term 1.
I20260620 23:17:09.825616 6748 mvcc.cc:205] Tried to move back new op lower bound from 7299061472415420416 to 7299061472336760832. Current Snapshot: MvccSnapshot[applied={T|T < 7299061472415420416}]
I20260620 23:17:09.825803 6725 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:17:09.826519 6725 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:17:09.829511 6692 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8cd64cba42949e3ad600fe220d17d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } } }
I20260620 23:17:09.829602 6692 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e [sys.catalog]: This master's current role is: LEADER
W20260620 23:17:09.830559 6746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:17:09.830621 6746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:17:09.830750 6735 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8cd64cba42949e3ad600fe220d17d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } } }
I20260620 23:17:09.830823 6735 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:09.830902 6735 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8cd64cba42949e3ad600fe220d17d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } } }
I20260620 23:17:09.830957 6735 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:09.831085 6750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8cd64cba42949e3ad600fe220d17d8e. Latest consensus state: current_term: 1 leader_uuid: "d8cd64cba42949e3ad600fe220d17d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cd64cba42949e3ad600fe220d17d8e" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 40419 } } peers { permanent_uuid: "b63bdb131a8041a5af926d73414baae1" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 39885 } } peers { permanent_uuid: "a059b8d0f84b4c0eb06da9639477a46e" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34721 } } }
I20260620 23:17:09.831161 6750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:09.845587 6725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8cd64cba42949e3ad600fe220d17d8e: Generated new TSK 0
I20260620 23:17:09.845826 6725 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:17:09.870637 6538 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:09.869098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.16.79.62:40419)
0620 23:17:09.869149 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.869151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.870000 (+ 849us) server_negotiation.cc:407] Connection header received
0620 23:17:09.870103 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.870104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.870115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.870145 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.870571 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60450: BlockingRecv error: recv got EOF from 127.0.0.1:60450 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 23:17:09.871701 6614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:09.869445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.16.79.61:39885)
0620 23:17:09.869483 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.869485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.870842 (+ 1357us) server_negotiation.cc:407] Connection header received
0620 23:17:09.871145 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.871146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.871158 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.871187 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.871644 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54328: BlockingRecv error: recv got EOF from 127.0.0.1:54328 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 23:17:09.872128 6690 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:09.869756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.16.79.60:34721)
0620 23:17:09.869800 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.869802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.871004 (+ 1202us) server_negotiation.cc:407] Connection header received
0620 23:17:09.871248 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.871249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.871259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.871287 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.872071 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39574: BlockingRecv error: recv got EOF from 127.0.0.1:39574 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 23:17:09.878250 6614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:09.877052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.16.79.61:39885)
0620 23:17:09.877087 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.877089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.877730 (+ 641us) server_negotiation.cc:407] Connection header received
0620 23:17:09.877835 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.877836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.877853 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.877883 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.878200 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54330: BlockingRecv error: recv got EOF from 127.0.0.1:54330 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 23:17:09.881188 6614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:09.880378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.16.79.61:39885)
0620 23:17:09.880410 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.880411 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.880647 (+ 236us) server_negotiation.cc:407] Connection header received
0620 23:17:09.880798 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.880799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.880809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.880837 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.881146 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54332: BlockingRecv error: recv got EOF from 127.0.0.1:54332 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 23:17:09.885665 6614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:09.884751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.16.79.61:39885)
0620 23:17:09.884789 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.884790 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.885122 (+ 332us) server_negotiation.cc:407] Connection header received
0620 23:17:09.885231 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.885232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.885242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.885276 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.885608 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54340: BlockingRecv error: recv got EOF from 127.0.0.1:54340 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 23:17:09.891733 6614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:09.891000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.16.79.61:39885)
0620 23:17:09.891032 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.891034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.891265 (+ 231us) server_negotiation.cc:407] Connection header received
0620 23:17:09.891353 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.891354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.891364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.891392 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.891691 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54350: BlockingRecv error: recv got EOF from 127.0.0.1:54350 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 23:17:09.901737 6614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:09.900893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54356 (local address 127.16.79.61:39885)
0620 23:17:09.900929 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.900931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.901184 (+ 253us) server_negotiation.cc:407] Connection header received
0620 23:17:09.901283 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.901284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.901294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.901323 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.901678 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54356: BlockingRecv error: recv got EOF from 127.0.0.1:54356 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 23:17:09.920111 6614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:09.919143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.16.79.61:39885)
0620 23:17:09.919186 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.919188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.919502 (+ 314us) server_negotiation.cc:407] Connection header received
0620 23:17:09.919601 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.919602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.919613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.919641 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.920056 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54366: BlockingRecv error: recv got EOF from 127.0.0.1:54366 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 23:17:09.954295 6614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:09.953401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.16.79.61:39885)
0620 23:17:09.953447 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:09.953449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:09.953703 (+ 254us) server_negotiation.cc:407] Connection header received
0620 23:17:09.953801 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:09.953802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:09.953813 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:09.953841 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:09.954233 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54378: BlockingRecv error: recv got EOF from 127.0.0.1:54378 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 23:17:10.020967 6614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:10.020051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.16.79.61:39885)
0620 23:17:10.020237 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:10.020239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:10.020251 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:17:10.020277 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:10.020278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:10.020288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:10.020351 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:10.020889 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54382: BlockingRecv error: recv got EOF from 127.0.0.1:54382 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
W20260620 23:17:10.150833 6614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:10.150140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.16.79.61:39885)
0620 23:17:10.150212 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:10.150215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:10.150288 (+ 73us) server_negotiation.cc:407] Connection header received
0620 23:17:10.150345 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:10.150345 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:10.150356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:10.150411 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:10.150743 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54388: BlockingRecv error: recv got EOF from 127.0.0.1:54388 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 23:17:10.409184 6614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:10.408232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.16.79.61:39885)
0620 23:17:10.408293 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:10.408298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:10.408311 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:17:10.408337 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:10.408337 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:10.408348 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:10.408418 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:10.409080 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54400: BlockingRecv error: recv got EOF from 127.0.0.1:54400 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:17:10.770820 6721 catalog_manager.cc:1259] Loaded cluster ID: 3770495f11fd489da1c930c27482b285
I20260620 23:17:10.770922 6721 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e: loading cluster ID for follower catalog manager: success
I20260620 23:17:10.771951 6721 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e: acquiring CA information for follower catalog manager: success
I20260620 23:17:10.772356 6721 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a059b8d0f84b4c0eb06da9639477a46e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:17:10.831192 6746 catalog_manager.cc:1259] Loaded cluster ID: 3770495f11fd489da1c930c27482b285
I20260620 23:17:10.831329 6746 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1: loading cluster ID for follower catalog manager: success
I20260620 23:17:10.832286 6746 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1: acquiring CA information for follower catalog manager: success
I20260620 23:17:10.832652 6746 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b63bdb131a8041a5af926d73414baae1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:17:10.925382 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:10.922620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.16.79.61:39885)
0620 23:17:10.922741 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:10.922743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:10.922749 (+ 6us) server_negotiation.cc:407] Connection header received
0620 23:17:10.922773 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:10.922774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:10.922833 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:10.922955 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:17:10.923143 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:10.923710 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:10.924485 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:10.924736 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:10.924823 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:10.925005 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:17:10.925126 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:17:10.925239 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:10.925266 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 23:17:10.925289 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":45,"threads_started":1}
I20260620 23:17:10.930372 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:10.927297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39588 (local address 127.16.79.60:34721)
0620 23:17:10.927399 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:10.927400 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:10.927451 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:17:10.927515 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:10.927515 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:10.927547 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:10.927616 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:17:10.927677 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:10.928160 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:10.929250 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:10.929535 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:10.929621 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:10.929859 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:17:10.930066 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:17:10.930153 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:10.930235 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 23:17:10.930261 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":48,"threads_started":1}
I20260620 23:17:10.934983 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:10.932284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.16.79.61:39885)
0620 23:17:10.932323 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:10.932325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:10.932444 (+ 119us) server_negotiation.cc:407] Connection header received
0620 23:17:10.932529 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:10.932531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:10.932539 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:10.932593 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:17:10.932724 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:10.933209 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:10.934150 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:10.934428 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:10.934506 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:10.934675 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:17:10.934706 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:17:10.934732 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:10.934861 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 23:17:10.934896 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:17:10.935088 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:10.932128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.16.79.60:34721)
0620 23:17:10.932177 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:10.932179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:10.932356 (+ 177us) server_negotiation.cc:407] Connection header received
0620 23:17:10.932465 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:10.932466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:10.932477 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:10.932528 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:17:10.932719 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:10.933439 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:10.934321 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:10.934594 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:10.934671 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:10.934816 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:17:10.934846 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:17:10.934894 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:10.934973 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 23:17:10.934998 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:17:10.935523 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:10.932097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.16.79.62:40419)
0620 23:17:10.932249 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:10.932251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:10.932335 (+ 84us) server_negotiation.cc:407] Connection header received
0620 23:17:10.932485 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:10.932486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:10.932531 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:10.932583 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:17:10.932842 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:10.933560 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:10.934491 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:10.934787 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:10.934852 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:10.935134 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:17:10.935237 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:17:10.935279 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:10.935417 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 23:17:10.935442 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":51,"threads_started":1}
I20260620 23:17:10.940868 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 6467
I20260620 23:17:10.945792 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 6542
I20260620 23:17:10.950724 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 6617
2026-06-20T23:17:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2844 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:17:11 dist-test-slave-pswb krb5kdc[6835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:17:11 dist-test-slave-pswb krb5kdc[6835](info): set up 2 sockets
Jun 20 23:17:11 dist-test-slave-pswb krb5kdc[6835](info): commencing operation
krb5kdc: starting...
W20260620 23:17:12.004982 16700 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/test-user.keytab.
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:17:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:17:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.79.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
Entry for principal kudu/127.16.79.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab.
I20260620 23:17:12.132637 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:33021
--webserver_interface=127.16.79.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:33021,127.16.79.61:46709,127.16.79.60:34249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:17:12.210239 6848 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:17:12.210449 6848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:17:12.210470 6848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:17:12.210491 6848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:17:12.211514 6848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:17:12.211555 6848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:17:12.211570 6848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:17:12.211581 6848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:17:12.211594 6848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:17:12.213019 6848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:33021,127.16.79.61:46709,127.16.79.60:34249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.62.keytab
--principal=kudu/127.16.79.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.62:33021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.79.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:17:12.213201 6848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:17:12.213361 6848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:17:12.218356 6854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:12.218355 6853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:12.218367 6856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:12.218595 6848 server_base.cc:1034] running on GCE node
I20260620 23:17:12.218827 6848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:17:12.219106 6848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:17:12.220271 6848 hybrid_clock.cc:648] HybridClock initialized: now 1781997432220251 us; error 40 us; skew 500 ppm
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:17:12.222981 6848 init.cc:377] Logged in from keytab as kudu/127.16.79.62@KRBTEST.COM (short username kudu)
I20260620 23:17:12.223445 6848 webserver.cc:460] Webserver started at http://127.16.79.62:35207/ using document root <none> and password file <none>
I20260620 23:17:12.223682 6848 fs_manager.cc:359] Metadata directory not provided
I20260620 23:17:12.223754 6848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:17:12.223924 6848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:17:12.225116 6848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/data/instance:
uuid: "151784230abe4204b4c58fd843acd710"
format_stamp: "Formatted at 2026-06-20 23:17:12 on dist-test-slave-pswb"
server_key: "5a87171795b8c38e36169622fa1e6ecd"
server_key_iv: "58131342ea2b2234f386767b9aab0d86"
server_key_version: "encryptionkey@0"
I20260620 23:17:12.225508 6848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/wal/instance:
uuid: "151784230abe4204b4c58fd843acd710"
format_stamp: "Formatted at 2026-06-20 23:17:12 on dist-test-slave-pswb"
server_key: "5a87171795b8c38e36169622fa1e6ecd"
server_key_iv: "58131342ea2b2234f386767b9aab0d86"
server_key_version: "encryptionkey@0"
I20260620 23:17:12.227098 6848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:17:12.227939 6863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:17:12.228271 6848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:17:12.228431 6848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/wal
uuid: "151784230abe4204b4c58fd843acd710"
format_stamp: "Formatted at 2026-06-20 23:17:12 on dist-test-slave-pswb"
server_key: "5a87171795b8c38e36169622fa1e6ecd"
server_key_iv: "58131342ea2b2234f386767b9aab0d86"
server_key_version: "encryptionkey@0"
I20260620 23:17:12.228531 6848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:17:12.269268 6848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:17:12.271575 6848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:17:12.271739 6848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:17:12.275806 6848 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.62:33021
I20260620 23:17:12.275854 6915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.62:33021 every 8 connection(s)
I20260620 23:17:12.276209 6848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/data/info.pb
I20260620 23:17:12.277295 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:17:12.277686 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 6848
I20260620 23:17:12.277808 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-0/wal/instance
I20260620 23:17:12.278019 16700 external_mini_cluster.cc:1428] Setting key 70ad3d3dbf92e9a41c3cbc08d03444e7
I20260620 23:17:12.278932 6916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.79.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.61@KRBTEST.COM" created.
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:17:12.292816 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.279722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.16.79.62:33021)
0620 23:17:12.279886 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.279895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.280005 (+ 110us) server_negotiation.cc:407] Connection header received
0620 23:17:12.280660 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.280662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.280678 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.280809 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.281584 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.282669 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.283559 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.284051 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.284168 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.289109 (+ 4941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.289116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.289117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.289124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.290794 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.291249 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.291251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.291251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.291324 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.291716 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.291718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.291718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.292049 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.292170 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.292489 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.292587 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260620 23:17:12.292841 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.279701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33021 (local address 127.0.0.1:40818)
0620 23:17:12.279867 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.279904 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:12.280519 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:12.280832 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:12.280835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:12.281200 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:12.281433 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.281440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.282778 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.282779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.283473 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.283475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.284238 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.284240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.284344 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.285082 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:12.285092 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:12.289019 (+ 3927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:12.290881 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.290888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.290890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.291164 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.291396 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.291397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.291397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.291574 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.292175 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:12.292182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:12.292272 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 23:17:12.292417 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 23:17:12.292688 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":91,"thread_start_us":96,"threads_started":1}
I20260620 23:17:12.293745 6916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:17:12.294112 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.293977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:46709 (local address 127.0.0.1:36634)
0620 23:17:12.294029 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.294082 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 23:17:12.294262 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.79.61:46709: connect: Connection refused (error 111)
I20260620 23:17:12.294705 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.294592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:46709 (local address 127.0.0.1:36636)
0620 23:17:12.294646 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.294676 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 23:17:12.294833 6916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.61:46709: Network error: Client connection negotiation failed: client connection to 127.16.79.61:46709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.61.keytab.
Entry for principal kudu/127.16.79.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.61.keytab.
I20260620 23:17:12.304577 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.61:46709
--webserver_interface=127.16.79.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.61.keytab
--principal=kudu/127.16.79.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:33021,127.16.79.61:46709,127.16.79.60:34249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:17:12.344129 6916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } attempt: 1
I20260620 23:17:12.344971 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.344759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:46709 (local address 127.0.0.1:36646)
0620 23:17:12.344849 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.344913 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:17:12.345510 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.345395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:46709 (local address 127.0.0.1:36654)
0620 23:17:12.345453 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.345487 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 23:17:12.345714 6916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.61:46709: Network error: Client connection negotiation failed: client connection to 127.16.79.61:46709: connect: Connection refused (error 111)
W20260620 23:17:12.384913 6923 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:17:12.385107 6923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:17:12.385124 6923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:17:12.385144 6923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:17:12.386253 6923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:17:12.386299 6923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:17:12.386313 6923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:17:12.386327 6923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:17:12.386338 6923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:17:12.387816 6923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:33021,127.16.79.61:46709,127.16.79.60:34249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.61.keytab
--principal=kudu/127.16.79.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.61:46709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.79.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:17:12.388041 6923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:17:12.388279 6923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:17:12.392984 6929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:12.393016 6928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:12.392998 6931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:12.393316 6923 server_base.cc:1034] running on GCE node
I20260620 23:17:12.393543 6923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:17:12.393759 6923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:17:12.394913 6923 hybrid_clock.cc:648] HybridClock initialized: now 1781997432394902 us; error 28 us; skew 500 ppm
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:17:12.397574 6923 init.cc:377] Logged in from keytab as kudu/127.16.79.61@KRBTEST.COM (short username kudu)
I20260620 23:17:12.398034 6923 webserver.cc:460] Webserver started at http://127.16.79.61:36541/ using document root <none> and password file <none>
I20260620 23:17:12.398274 6923 fs_manager.cc:359] Metadata directory not provided
I20260620 23:17:12.398353 6923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:17:12.398520 6923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:17:12.399626 6923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/data/instance:
uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9"
format_stamp: "Formatted at 2026-06-20 23:17:12 on dist-test-slave-pswb"
server_key: "d18e36feb15b16a18da551d79c0762d9"
server_key_iv: "badded943d7ad3c0a27923d29ead691e"
server_key_version: "encryptionkey@0"
I20260620 23:17:12.399999 6923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/wal/instance:
uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9"
format_stamp: "Formatted at 2026-06-20 23:17:12 on dist-test-slave-pswb"
server_key: "d18e36feb15b16a18da551d79c0762d9"
server_key_iv: "badded943d7ad3c0a27923d29ead691e"
server_key_version: "encryptionkey@0"
I20260620 23:17:12.401376 6923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260620 23:17:12.402108 6938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:17:12.402273 6923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:17:12.402374 6923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/wal
uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9"
format_stamp: "Formatted at 2026-06-20 23:17:12 on dist-test-slave-pswb"
server_key: "d18e36feb15b16a18da551d79c0762d9"
server_key_iv: "badded943d7ad3c0a27923d29ead691e"
server_key_version: "encryptionkey@0"
I20260620 23:17:12.402455 6923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:17:12.413909 6916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } attempt: 2
I20260620 23:17:12.414341 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.414213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:46709 (local address 127.0.0.1:36666)
0620 23:17:12.414273 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.414305 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:17:12.414718 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.414614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:46709 (local address 127.0.0.1:36668)
0620 23:17:12.414661 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.414693 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.61:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 23:17:12.414851 6916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.61:46709: Network error: Client connection negotiation failed: client connection to 127.16.79.61:46709: connect: Connection refused (error 111)
I20260620 23:17:12.430531 6923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:17:12.433374 6923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:17:12.433529 6923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:17:12.437793 6923 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.61:46709
I20260620 23:17:12.437912 6990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.61:46709 every 8 connection(s)
I20260620 23:17:12.438207 6923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/data/info.pb
I20260620 23:17:12.439328 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 6923
I20260620 23:17:12.439422 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-1/wal/instance
I20260620 23:17:12.439468 6991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:17:12.439710 16700 external_mini_cluster.cc:1428] Setting key fba41cd49b713c8ba78f7bfdb62d48f3
I20260620 23:17:12.441221 6991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.79.60@KRBTEST.COM; defaulting to no policy
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.61@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.79.60@KRBTEST.COM" created.
I20260620 23:17:12.450337 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.441994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33021 (local address 127.0.0.1:40820)
0620 23:17:12.442196 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.442242 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:12.442701 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:12.442979 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:12.442982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:12.443209 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:12.443388 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.443391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.444078 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.444079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.444902 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.444904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.445344 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.445345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.445465 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.445942 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:12.445950 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:12.447263 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:12.448901 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.448908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.448910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.449157 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.449355 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.449355 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.449356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.449499 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.449802 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:12.449808 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:12.449966 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 23:17:12.450072 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:17:12.450187 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260620 23:17:12.450367 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.442062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.16.79.62:33021)
0620 23:17:12.442152 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.442155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.442331 (+ 176us) server_negotiation.cc:407] Connection header received
0620 23:17:12.442805 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.442806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.442818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.442888 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.443490 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.443986 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.445017 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.445225 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.445313 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.447359 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.447360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.447361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.447368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.448782 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.449234 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.449235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.449236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.449290 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.449613 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.449613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.449614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.449723 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.449811 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.450148 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.450241 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:17:12.450917 6991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.61@KRBTEST.COM for kudu/127.16.79.61@KRBTEST.COM
I20260620 23:17:12.458482 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.451277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:46709 (local address 127.0.0.1:36680)
0620 23:17:12.451326 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.451337 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:12.451374 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:12.451633 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:12.451634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:12.451769 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:12.451806 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.451807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.452661 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.452661 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.453250 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.453252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.453716 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.453716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.453777 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.454203 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:12.454206 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:12.455516 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:12.457545 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.457546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.457547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.457699 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.457909 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.457909 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.457909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.457958 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.458236 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:12.458237 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:12.458288 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:17:12.458344 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 23:17:12.458426 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 23:17:12.458588 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.451361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.16.79.61:46709)
0620 23:17:12.451505 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.451508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.451524 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:17:12.451540 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.451541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.451550 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.451634 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.451913 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.452553 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.453338 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.453628 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.453716 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.455606 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.455613 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.455614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.455622 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.457422 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.457778 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.457780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.457781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.457840 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.458014 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.458014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.458015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.458135 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.458238 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.458386 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.458482 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260620 23:17:12.459157 6991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:17:12.459450 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.459353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51782)
0620 23:17:12.459383 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.459432 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 23:17:12.459602 6939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
I20260620 23:17:12.460100 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.460006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51792)
0620 23:17:12.460052 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.460080 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 23:17:12.460234 6991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:34249: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.79.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.60.keytab.
Entry for principal kudu/127.16.79.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.60.keytab.
I20260620 23:17:12.466069 16700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmNabNW/build/release/bin/kudu
/tmp/dist-test-taskmNabNW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.60:34249
--webserver_interface=127.16.79.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.60.keytab
--principal=kudu/127.16.79.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.79.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.79.62:33021,127.16.79.61:46709,127.16.79.60:34249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:17:12.506047 6916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } attempt: 3
I20260620 23:17:12.509469 6991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } attempt: 1
I20260620 23:17:12.509914 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.509724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51794)
0620 23:17:12.509801 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.509867 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:17:12.510402 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.510272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51808)
0620 23:17:12.510347 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.510372 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":42}
W20260620 23:17:12.510491 6991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:34249: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for kudu/127.16.79.61@KRBTEST.COM
I20260620 23:17:12.514676 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.506439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:46709 (local address 127.0.0.1:36696)
0620 23:17:12.506535 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.506548 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:12.506601 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:12.506776 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:12.506776 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:12.506922 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:12.506988 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.506989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.507925 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.507926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.508465 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.508467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.509056 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.509058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.509174 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.509915 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:12.509920 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:12.511575 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:12.513659 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.513660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.513660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.513870 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.514077 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.514077 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.514078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.514131 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.514415 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:12.514416 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:12.514460 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 23:17:12.514528 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 23:17:12.514608 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 23:17:12.514736 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.506566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.16.79.61:46709)
0620 23:17:12.506628 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.506630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.506641 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:17:12.506670 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.506671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.506685 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.506763 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.507059 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.507831 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.508551 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.508908 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.508991 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.511686 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.511687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.511688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.511696 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.513494 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.513947 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.513948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.513948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.514010 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.514189 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.514190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.514190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.514326 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.514392 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.514569 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.514646 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:17:12.515308 6916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:17:12.515664 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.515559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51822)
0620 23:17:12.515609 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.515636 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 23:17:12.516139 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.515996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51824)
0620 23:17:12.516098 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.516124 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260620 23:17:12.516259 6916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:34249: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
I20260620 23:17:12.547457 6916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } attempt: 1
I20260620 23:17:12.548130 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.547963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51828)
0620 23:17:12.548043 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.548093 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 23:17:12.548828 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.548588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51840)
0620 23:17:12.548766 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.548803 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 23:17:12.548964 6916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:34249: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
W20260620 23:17:12.551326 6998 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:17:12.551558 6998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:17:12.551595 6998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:17:12.551633 6998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:17:12.552804 6998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:17:12.552861 6998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:17:12.552886 6998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:17:12.552913 6998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:17:12.552939 6998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:17:12.554381 6998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.79.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.79.62:33021,127.16.79.61:46709,127.16.79.60:34249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/krb5kdc/kudu_127.16.79.60.keytab
--principal=kudu/127.16.79.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.79.60:34249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.79.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:11:48 UTC on ccycloud.quasar-qwxqme.root.comops.site
build id 4457789
I20260620 23:17:12.554651 6998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:17:12.554881 6998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:17:12.559688 7004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:12.559732 7006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:17:12.559757 7003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:17:12.560060 6998 server_base.cc:1034] running on GCE node
I20260620 23:17:12.560303 6998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:17:12.560920 6998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:17:12.562104 6998 hybrid_clock.cc:648] HybridClock initialized: now 1781997432562078 us; error 46 us; skew 500 ppm
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:17:12.564594 6998 init.cc:377] Logged in from keytab as kudu/127.16.79.60@KRBTEST.COM (short username kudu)
I20260620 23:17:12.565104 6998 webserver.cc:460] Webserver started at http://127.16.79.60:41707/ using document root <none> and password file <none>
I20260620 23:17:12.565347 6998 fs_manager.cc:359] Metadata directory not provided
I20260620 23:17:12.565397 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:17:12.565502 6998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:17:12.566488 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/data/instance:
uuid: "354a4d390e794cac85bb2c37d3f8fd09"
format_stamp: "Formatted at 2026-06-20 23:17:12 on dist-test-slave-pswb"
server_key: "b8c253fc22fbe1a8826232aa4c9b52d3"
server_key_iv: "b1ac5ab58b4195933328f98cb5fd1b96"
server_key_version: "encryptionkey@0"
I20260620 23:17:12.566836 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/wal/instance:
uuid: "354a4d390e794cac85bb2c37d3f8fd09"
format_stamp: "Formatted at 2026-06-20 23:17:12 on dist-test-slave-pswb"
server_key: "b8c253fc22fbe1a8826232aa4c9b52d3"
server_key_iv: "b1ac5ab58b4195933328f98cb5fd1b96"
server_key_version: "encryptionkey@0"
I20260620 23:17:12.568182 6998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:17:12.569334 7013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:17:12.569621 6998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:17:12.569729 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/data,/tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/wal
uuid: "354a4d390e794cac85bb2c37d3f8fd09"
format_stamp: "Formatted at 2026-06-20 23:17:12 on dist-test-slave-pswb"
server_key: "b8c253fc22fbe1a8826232aa4c9b52d3"
server_key_iv: "b1ac5ab58b4195933328f98cb5fd1b96"
server_key_version: "encryptionkey@0"
I20260620 23:17:12.569814 6998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:17:12.578648 6991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } attempt: 2
I20260620 23:17:12.579053 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.578926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51854)
0620 23:17:12.578984 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.579021 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 23:17:12.579402 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.579312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51856)
0620 23:17:12.579357 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.579383 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 23:17:12.579532 6991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.79.60:34249: Network error: Client connection negotiation failed: client connection to 127.16.79.60:34249: connect: Connection refused (error 111)
I20260620 23:17:12.584578 6998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:17:12.586937 6998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:17:12.587105 6998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:17:12.592504 6998 rpc_server.cc:307] RPC server started. Bound to: 127.16.79.60:34249
I20260620 23:17:12.592573 7065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.79.60:34249 every 8 connection(s)
I20260620 23:17:12.593062 6998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/data/info.pb
I20260620 23:17:12.594244 7066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:17:12.596380 7066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:17:12.601713 16700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmNabNW/build/release/bin/kudu as pid 6998
I20260620 23:17:12.601821 16700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmNabNW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997163808439-16700-0/minicluster-data/master-2/wal/instance
I20260620 23:17:12.602034 16700 external_mini_cluster.cc:1428] Setting key 92e879d608d1cb82a848188066b178f9
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.60@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
I20260620 23:17:12.606292 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.597085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.16.79.62:33021)
0620 23:17:12.597157 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.597160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.597406 (+ 246us) server_negotiation.cc:407] Connection header received
0620 23:17:12.597826 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.597827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.597837 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.597889 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.598590 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.599080 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.600144 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.600395 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.600501 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.603190 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.603192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.603192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.603201 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.604802 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.605249 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.605249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.605250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.605312 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.605606 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.605606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.605607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.605753 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.605827 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.606095 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.606163 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:17:12.606458 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.597071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.62:33021 (local address 127.0.0.1:40832)
0620 23:17:12.597282 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.597323 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:12.597743 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:12.597949 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:12.597952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:12.598180 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:12.598369 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.598372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.599216 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.599217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.599977 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.599979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.600533 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.600534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.600642 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.601127 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:12.601135 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:12.603078 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:12.604876 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.604883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.604891 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.605179 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.605380 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.605380 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.605381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.605521 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.605820 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:12.605825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:12.605932 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 23:17:12.606053 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 23:17:12.606217 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":132,"threads_started":1}
I20260620 23:17:12.607060 7066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.60@KRBTEST.COM
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.60@KRBTEST.COM for kudu/127.16.79.61@KRBTEST.COM
I20260620 23:17:12.611909 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.602464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.16.79.60:34249)
0620 23:17:12.602769 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.602774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.602797 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:17:12.602815 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.602816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.602833 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.602942 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.603699 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.604361 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.605497 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.605927 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.605979 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.608549 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.608558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.608558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.608564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.610276 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.610647 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.610650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.610650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.610716 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.610918 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.610919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.610919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.611081 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.611227 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.611704 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.611784 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":158,"threads_started":1}
I20260620 23:17:12.614377 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.607319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.61:46709 (local address 127.0.0.1:36700)
0620 23:17:12.607433 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.607444 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:12.607496 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:12.607620 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:12.607621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:12.607757 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:12.607799 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.607801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.608510 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.608511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.609074 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.609075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.609452 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.609453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.609524 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.609838 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:12.609842 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:12.610940 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:12.613062 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.613063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.613063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.613230 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.613480 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.613481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.613481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.613547 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.614152 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:12.614152 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:12.614185 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:17:12.614238 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 23:17:12.614303 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 23:17:12.614771 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.607387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.16.79.61:46709)
0620 23:17:12.607437 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.607439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.607481 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:17:12.607557 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.607558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.607566 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.607604 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.607882 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.608430 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.609168 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.609372 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.609454 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.611001 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.611002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.611002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.611009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.612985 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.613306 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.613307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.613308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.613368 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.613714 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.613716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.613717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.613907 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.614247 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.614295 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.614376 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:17:12.615116 7066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.62@KRBTEST.COM
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.60@KRBTEST.COM for kudu/127.16.79.60@KRBTEST.COM
I20260620 23:17:12.620599 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.612652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.16.79.62:33021)
0620 23:17:12.612686 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.612727 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.612734 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:17:12.612758 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.612758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.612767 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.612822 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.613355 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.613913 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.615036 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.615325 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.615382 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.617704 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.617707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.617707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.617719 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.619431 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.619827 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.619828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.619829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.619888 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.620091 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.620092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.620093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.620203 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.620368 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.620415 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.620484 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:17:12.622786 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.615490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51880)
0620 23:17:12.615543 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.615553 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:12.615590 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:12.615922 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:12.615923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:12.616051 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:12.616075 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.616076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.617025 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.617026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.617630 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.617651 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.618075 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.618076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.618125 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.618416 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:12.618421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:12.619811 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:12.621755 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.621756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.621756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.621962 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.622155 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.622156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.622156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.622220 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.622435 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:12.622435 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:12.622476 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:17:12.622548 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 23:17:12.622642 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:17:12.622834 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.615558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.16.79.60:34249)
0620 23:17:12.615823 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.615824 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.615830 (+ 6us) server_negotiation.cc:407] Connection header received
0620 23:17:12.615844 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.615845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.615852 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.615919 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.616164 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.616884 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.617759 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.617964 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.618056 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.619911 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.619912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.619912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.619920 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.621662 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.622044 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.622045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.622045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.622092 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.622289 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.622290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.622290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.622377 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.622422 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.622564 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.622704 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":249}
I20260620 23:17:12.624115 6916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } attempt: 2
I20260620 23:17:12.624414 7066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09: Bootstrap starting.
I20260620 23:17:12.625286 7066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09: Neither blocks nor log segments found. Creating new log.
I20260620 23:17:12.625624 7066 log.cc:824] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.79.61@KRBTEST.COM
I20260620 23:17:12.626441 7066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09: No bootstrap required, opened a new log
I20260620 23:17:12.628156 7066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } }
I20260620 23:17:12.628317 7066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354a4d390e794cac85bb2c37d3f8fd09, State: Initialized, Role: FOLLOWER
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.62@KRBTEST.COM for kudu/127.16.79.60@KRBTEST.COM
I20260620 23:17:12.628394 7066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } }
I20260620 23:17:12.628741 7066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:17:12.628859 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } } }
I20260620 23:17:12.629015 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:17:12.630621 7083 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 23:17:12.630658 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.621202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.16.79.61:46709)
0620 23:17:12.621238 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.621239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.621250 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:17:12.621330 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.621331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.621338 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.621375 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.621953 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.622646 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.623518 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.623771 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.623855 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.626549 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.626550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.626550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.626557 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.628898 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.629258 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.629259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.629259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.629315 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.629820 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.629821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.629821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.629950 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.630089 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.630480 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.630551 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 23:17:12.630889 7083 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:17:12.631781 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.624326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51886)
0620 23:17:12.624409 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.624418 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:12.624454 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:12.624599 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:12.624600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:12.624809 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:12.624855 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.624857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.625767 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.625768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.626594 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.626597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.627037 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.627038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.627102 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.627449 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:12.627454 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:12.628725 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:12.630612 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.630613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.630614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.630883 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.631092 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.631093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.631093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.631165 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.631486 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:12.631487 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:12.631529 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:17:12.631591 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:17:12.631676 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 23:17:12.632040 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.624414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.16.79.60:34249)
0620 23:17:12.624453 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.624454 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.624463 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:17:12.624493 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.624494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.624503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.624935 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.624967 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.625668 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.626672 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.626965 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.627037 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.628822 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.628823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.628824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.628831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.630534 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.630971 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.630972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.630972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.631033 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.631242 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.631242 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.631243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.631389 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.631457 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.631887 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.631955 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:17:12.633531 6916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710: Bootstrap starting.
I20260620 23:17:12.630928 6998 master_runner.cc:186] Error getting master registration for 127.16.79.62:33021: OK, instance_id { permanent_uuid: "151784230abe4204b4c58fd843acd710" instance_seqno: 1781997432274737 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:17:12.634027 6998 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:17:12.634112 16700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:17:12.634132 6916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710: Neither blocks nor log segments found. Creating new log.
I20260620 23:17:12.634402 6916 log.cc:824] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710: Log is configured to *not* fsync() on all Append() calls
I20260620 23:17:12.634924 6890 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "151784230abe4204b4c58fd843acd710"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40842
I20260620 23:17:12.635878 6916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710: No bootstrap required, opened a new log
I20260620 23:17:12.636988 6916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } }
I20260620 23:17:12.637112 6916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151784230abe4204b4c58fd843acd710, State: Initialized, Role: FOLLOWER
I20260620 23:17:12.637216 6916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } }
I20260620 23:17:12.637557 6916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:17:12.637523 7086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } } }
I20260620 23:17:12.637610 7086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:12.637797 6890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:17:12.637854 6890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:17:12.638705 6890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } }
I20260620 23:17:12.638944 6890 leader_election.cc:288] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [CANDIDATE]: Term 1 election: Requested vote from peers 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 (127.16.79.61:46709), 354a4d390e794cac85bb2c37d3f8fd09 (127.16.79.60:34249)
I20260620 23:17:12.639168 7040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "151784230abe4204b4c58fd843acd710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "354a4d390e794cac85bb2c37d3f8fd09"
I20260620 23:17:12.639288 7040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:17:12.639391 7097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:17:12.639432 7097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:17:12.640121 7040 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151784230abe4204b4c58fd843acd710 in term 1.
I20260620 23:17:12.640676 6864 leader_election.cc:302] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151784230abe4204b4c58fd843acd710, 354a4d390e794cac85bb2c37d3f8fd09; no voters:
I20260620 23:17:12.640836 7086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:17:12.640992 7086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [term 1 LEADER]: Becoming Leader. State: Replica: 151784230abe4204b4c58fd843acd710, State: Running, Role: LEADER
I20260620 23:17:12.641076 7086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } }
I20260620 23:17:12.641446 7086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 151784230abe4204b4c58fd843acd710. Latest consensus state: current_term: 1 leader_uuid: "151784230abe4204b4c58fd843acd710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } } }
I20260620 23:17:12.641525 7086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:12.641781 7107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:17:12.642074 7107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:17:12.640421 6848 master_runner.cc:186] Error getting master registration for 127.16.79.61:46709: OK, instance_id { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" instance_seqno: 1781997432436585 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:17:12.642802 6848 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:17:12.644776 7040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [term 1 FOLLOWER]: Refusing update from remote peer 151784230abe4204b4c58fd843acd710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:17:12.645206 7105 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:17:12.646525 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 151784230abe4204b4c58fd843acd710. Latest consensus state: current_term: 1 leader_uuid: "151784230abe4204b4c58fd843acd710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } } }
I20260620 23:17:12.646631 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:12.648301 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.640194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.16.79.62:33021)
0620 23:17:12.640231 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.640233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.640535 (+ 302us) server_negotiation.cc:407] Connection header received
0620 23:17:12.640634 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.640635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.640647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.640765 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.641787 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.642474 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.643506 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.643792 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.643854 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.645395 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.645396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.645396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.645403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.647098 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.647475 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.647476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.647476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.647524 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.647717 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.647718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.647719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.647841 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.647933 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.648144 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.648220 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:17:12.648417 7086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "151784230abe4204b4c58fd843acd710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } } }
I20260620 23:17:12.648489 7086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [sys.catalog]: This master's current role is: LEADER
I20260620 23:17:12.648811 7109 mvcc.cc:205] Tried to move back new op lower bound from 7299061484110368768 to 7299061484098461696. Current Snapshot: MvccSnapshot[applied={T|T < 7299061484110368768}]
I20260620 23:17:12.648998 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "151784230abe4204b4c58fd843acd710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } } }
I20260620 23:17:12.649089 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:12.649438 7110 mvcc.cc:205] Tried to move back new op lower bound from 7299061484110368768 to 7299061484098461696. Current Snapshot: MvccSnapshot[applied={T|T < 7299061484110368768}]
I20260620 23:17:12.650942 7107 catalog_manager.cc:1347] Generated new cluster ID: b8754f33997c42f99a1b2c8f4d467878
I20260620 23:17:12.650988 7107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:17:12.655709 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.643061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.16.79.60:34249)
0620 23:17:12.643099 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.643100 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.648615 (+ 5515us) server_negotiation.cc:407] Connection header received
0620 23:17:12.648728 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.648729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.648738 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.648767 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.649695 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.650392 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.651544 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.651862 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.651935 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.653149 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.653150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.653150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.653156 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.654595 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.655044 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.655045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.655046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.655090 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.655244 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.655244 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.655244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.655345 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.655405 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.655570 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.655631 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:17:12.656546 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.640274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.16.79.61:46709)
0620 23:17:12.640342 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.640343 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.643934 (+ 3591us) server_negotiation.cc:407] Connection header received
0620 23:17:12.644031 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.644032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.644041 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.644070 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.645655 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.646361 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.647345 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.647682 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.647741 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.650576 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.650578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.650578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.650585 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.654570 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.655307 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.655308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.655308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.655382 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.655731 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.655732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.655732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.655860 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.655942 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.656379 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.656457 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:17:12.670657 6991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } attempt: 3
I20260620 23:17:12.674136 7107 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:17:12.674829 7107 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 23:17:12 dist-test-slave-pswb krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997432, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.79.61@KRBTEST.COM for kudu/127.16.79.60@KRBTEST.COM
I20260620 23:17:12.678886 7107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710: Generated new TSK 0
I20260620 23:17:12.678972 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.670890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.79.60:34249 (local address 127.0.0.1:51906)
0620 23:17:12.670963 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 23:17:12.670974 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:17:12.671008 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:17:12.671243 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:17:12.671244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:17:12.671402 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:17:12.671443 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.671445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.672321 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.672322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.673199 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:17:12.673202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.673813 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.673814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:17:12.673868 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.674387 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:17:12.674392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:17:12.676016 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:17:12.677839 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.677840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.677840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.678054 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.678299 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:17:12.678299 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:17:12.678300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:17:12.678368 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:17:12.678674 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:17:12.678675 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:17:12.678728 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 23:17:12.678789 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:17:12.678877 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 23:17:12.679040 7107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:17:12.679579 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:12.671084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.16.79.60:34249)
0620 23:17:12.671116 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.671118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.671124 (+ 6us) server_negotiation.cc:407] Connection header received
0620 23:17:12.671142 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.671143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.671151 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.671204 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.671516 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.672235 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.673398 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:12.673704 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:12.673763 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:12.676104 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:17:12.676105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:17:12.676105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:17:12.676111 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:17:12.677731 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.678144 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.678145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.678145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.678218 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:17:12.678445 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:17:12.678445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:17:12.678446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:17:12.678577 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:17:12.678633 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:12.679416 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 23:17:12.679489 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:17:12.681010 6991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9: Bootstrap starting.
I20260620 23:17:12.681704 6991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9: Neither blocks nor log segments found. Creating new log.
I20260620 23:17:12.681986 6991 log.cc:824] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:17:12.691747 6991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9: No bootstrap required, opened a new log
I20260620 23:17:12.693784 6991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } }
I20260620 23:17:12.693903 6991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc3ad6f5c5b4dab85ca0f7cc33c15a9, State: Initialized, Role: FOLLOWER
I20260620 23:17:12.693974 6991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } }
I20260620 23:17:12.694265 6991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:17:12.695647 6964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:17:12.696506 6964 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [term 1 FOLLOWER]: Refusing update from remote peer 151784230abe4204b4c58fd843acd710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:17:12.696864 7086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:17:12.698261 6965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "151784230abe4204b4c58fd843acd710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9"
I20260620 23:17:12.698855 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "151784230abe4204b4c58fd843acd710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } } }
I20260620 23:17:12.698990 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:12.699081 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 151784230abe4204b4c58fd843acd710. Latest consensus state: current_term: 1 leader_uuid: "151784230abe4204b4c58fd843acd710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } } }
I20260620 23:17:12.699169 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:12.699237 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "151784230abe4204b4c58fd843acd710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } } }
I20260620 23:17:12.699424 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:17:12.700070 6965 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 151784230abe4204b4c58fd843acd710 for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260620 23:17:12.701418 7128 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:17:12.701481 7128 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:17:12.701766 7130 mvcc.cc:205] Tried to move back new op lower bound from 7299061484248084480 to 7299061484098461696. Current Snapshot: MvccSnapshot[applied={T|T < 7299061484110368768}]
W20260620 23:17:12.702658 6920 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:12.700315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.16.79.62:33021)
0620 23:17:12.700352 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.700354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.702027 (+ 1673us) server_negotiation.cc:407] Connection header received
0620 23:17:12.702126 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.702127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.702138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.702167 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.702601 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40866: BlockingRecv error: recv got EOF from 127.0.0.1:40866 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 23:17:12.703434 6996 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:12.700644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.16.79.61:46709)
0620 23:17:12.700678 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.700679 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.702929 (+ 2250us) server_negotiation.cc:407] Connection header received
0620 23:17:12.703022 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.703023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.703033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.703100 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.703386 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36722: BlockingRecv error: recv got EOF from 127.0.0.1:36722 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 23:17:12.703994 7070 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:12.701030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.16.79.60:34249)
0620 23:17:12.701131 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.701133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.703366 (+ 2233us) server_negotiation.cc:407] Connection header received
0620 23:17:12.703468 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.703469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.703479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.703506 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.703939 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51910: BlockingRecv error: recv got EOF from 127.0.0.1:51910 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 23:17:12.714895 6996 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:12.714074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.16.79.61:46709)
0620 23:17:12.714124 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.714126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.714324 (+ 198us) server_negotiation.cc:407] Connection header received
0620 23:17:12.714446 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.714447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.714465 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.714496 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.714841 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36732: BlockingRecv error: recv got EOF from 127.0.0.1:36732 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 23:17:12.718024 6996 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:12.717290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.16.79.61:46709)
0620 23:17:12.717321 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.717323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.717555 (+ 232us) server_negotiation.cc:407] Connection header received
0620 23:17:12.717646 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.717647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.717656 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.717684 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.717981 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36734: BlockingRecv error: recv got EOF from 127.0.0.1:36734 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 23:17:12.721997 6996 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:12.721185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.16.79.61:46709)
0620 23:17:12.721216 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.721218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.721538 (+ 320us) server_negotiation.cc:407] Connection header received
0620 23:17:12.721632 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.721633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.721642 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.721669 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.721958 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36748: BlockingRecv error: recv got EOF from 127.0.0.1:36748 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 23:17:12.727838 6996 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:12.727121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.16.79.61:46709)
0620 23:17:12.727152 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.727153 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.727361 (+ 208us) server_negotiation.cc:407] Connection header received
0620 23:17:12.727470 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.727471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.727481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.727508 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.727799 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36752: BlockingRecv error: recv got EOF from 127.0.0.1:36752 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 23:17:12.737905 6996 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:12.737139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.16.79.61:46709)
0620 23:17:12.737172 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.737173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.737417 (+ 244us) server_negotiation.cc:407] Connection header received
0620 23:17:12.737506 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.737507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.737517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.737545 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.737861 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36766: BlockingRecv error: recv got EOF from 127.0.0.1:36766 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 23:17:12.755925 6996 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:12.755119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.16.79.61:46709)
0620 23:17:12.755154 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.755156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.755399 (+ 243us) server_negotiation.cc:407] Connection header received
0620 23:17:12.755496 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.755497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.755507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.755536 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.755867 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36774: BlockingRecv error: recv got EOF from 127.0.0.1:36774 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 23:17:12.790479 6996 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:12.789593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.16.79.61:46709)
0620 23:17:12.789641 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.789643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.789915 (+ 272us) server_negotiation.cc:407] Connection header received
0620 23:17:12.790015 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.790016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.790028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.790058 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.790420 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36786: BlockingRecv error: recv got EOF from 127.0.0.1:36786 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 23:17:12.856722 6996 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:12.855972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.16.79.61:46709)
0620 23:17:12.856015 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.856018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.856130 (+ 112us) server_negotiation.cc:407] Connection header received
0620 23:17:12.856211 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.856212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.856223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.856294 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.856623 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36798: BlockingRecv error: recv got EOF from 127.0.0.1:36798 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 23:17:12.986711 6996 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:12.985934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.16.79.61:46709)
0620 23:17:12.986030 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:12.986033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:12.986137 (+ 104us) server_negotiation.cc:407] Connection header received
0620 23:17:12.986238 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:12.986239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:12.986252 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:12.986326 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:12.986653 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36810: BlockingRecv error: recv got EOF from 127.0.0.1:36810 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:17:13.130190 7086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "151784230abe4204b4c58fd843acd710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151784230abe4204b4c58fd843acd710" member_type: VOTER last_known_addr { host: "127.16.79.62" port: 33021 } } peers { permanent_uuid: "4fc3ad6f5c5b4dab85ca0f7cc33c15a9" member_type: VOTER last_known_addr { host: "127.16.79.61" port: 46709 } } peers { permanent_uuid: "354a4d390e794cac85bb2c37d3f8fd09" member_type: VOTER last_known_addr { host: "127.16.79.60" port: 34249 } } }
I20260620 23:17:13.130395 7086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 151784230abe4204b4c58fd843acd710 [sys.catalog]: This master's current role is: LEADER
W20260620 23:17:13.244805 6996 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:17:13.244096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.16.79.61:46709)
0620 23:17:13.244170 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:13.244172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:13.244183 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:17:13.244255 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:13.244256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:13.244268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:13.244352 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:17:13.244756 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36814: BlockingRecv error: recv got EOF from 127.0.0.1:36814 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:17:13.631368 7083 catalog_manager.cc:1259] Loaded cluster ID: b8754f33997c42f99a1b2c8f4d467878
I20260620 23:17:13.631474 7083 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09: loading cluster ID for follower catalog manager: success
I20260620 23:17:13.632457 7083 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09: acquiring CA information for follower catalog manager: success
I20260620 23:17:13.632822 7083 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 354a4d390e794cac85bb2c37d3f8fd09: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:17:13.701923 7128 catalog_manager.cc:1259] Loaded cluster ID: b8754f33997c42f99a1b2c8f4d467878
I20260620 23:17:13.702004 7128 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9: loading cluster ID for follower catalog manager: success
I20260620 23:17:13.702967 7128 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9: acquiring CA information for follower catalog manager: success
I20260620 23:17:13.703284 7128 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fc3ad6f5c5b4dab85ca0f7cc33c15a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:17:13.760950 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:13.758162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.16.79.61:46709)
0620 23:17:13.758267 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:13.758269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:13.758283 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:17:13.758365 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:13.758366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:13.758412 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:13.758483 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:17:13.758700 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:13.759209 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:13.760022 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:13.760212 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:13.760267 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:13.760416 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:17:13.760517 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:17:13.760575 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:13.760671 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 23:17:13.760834 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":62,"threads_started":1}
I20260620 23:17:13.765093 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:13.762678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.16.79.60:34249)
0620 23:17:13.762799 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:13.762801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:13.762810 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:17:13.762905 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:13.762906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:13.762945 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:13.763022 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:17:13.763123 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:13.763589 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:13.764302 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:13.764512 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:13.764600 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:13.764804 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:17:13.764918 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:17:13.764984 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:13.765023 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 23:17:13.765039 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":48,"threads_started":1}
I20260620 23:17:13.769248 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:13.766885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.16.79.60:34249)
0620 23:17:13.766928 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:13.766931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:13.767029 (+ 98us) server_negotiation.cc:407] Connection header received
0620 23:17:13.767122 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:13.767123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:13.767131 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:13.767179 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:17:13.767292 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:13.767730 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:13.768490 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:13.768672 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:13.768798 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:13.768948 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:17:13.768981 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:17:13.769041 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:13.769171 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 23:17:13.769189 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:17:13.769822 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:13.766963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.16.79.61:46709)
0620 23:17:13.767003 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:13.767005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:13.767071 (+ 66us) server_negotiation.cc:407] Connection header received
0620 23:17:13.767134 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:13.767135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:13.767145 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:13.767191 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:17:13.767339 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:13.768027 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:13.769079 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:13.769373 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:13.769447 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:13.769587 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:17:13.769619 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:17:13.769673 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:13.769728 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 23:17:13.769752 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:17:13.770155 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:17:13.766657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40880 (local address 127.16.79.62:33021)
0620 23:17:13.766792 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 23:17:13.766794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:17:13.766805 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:17:13.766893 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:17:13.766894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:17:13.766943 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:17:13.767011 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:17:13.767168 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:13.767936 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:13.769064 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:17:13.769564 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:17:13.769620 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:17:13.769831 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:17:13.769932 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:17:13.769994 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:17:13.770054 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 23:17:13.770074 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":55,"threads_started":1}
I20260620 23:17:13.772161 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 6848
I20260620 23:17:13.778005 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 6923
I20260620 23:17:13.782735 16700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmNabNW/build/release/bin/kudu with pid 6998
2026-06-20T23:17:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2832 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5677 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (269982 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST